our api docs are a disaster

Brandon | Cloudflare TSEOP•10/17/23, 11:09 PM

i wish there was a way to query the api endpoint like

Brandon | Cloudflare TSEOP•10/17/23, 11:09 PM

curl -x get https://api.cloudflare.com/client/v4/curl -x get https://api.cloudflare.com/client/v4/

and just have it poop out all the endpoints

James•10/17/23, 11:10 PM

it's the rulesets api now, sec

James•10/17/23, 11:10 PM

https://developers.cloudflare.com/ruleset-engine/rulesets-api/

BBrandon | Cloudflare TSE our api docs are a disaster

James•10/17/23, 11:11 PM

private thread, sorry everyone else, but yes, you might find this an insightful read: https://canary.discord.com/channels/595317990191398933/1042449111896555530

James•10/17/23, 11:11 PM

https://developers.cloudflare.com/waf/rate-limiting-rules/create-api/ is also a good reference

BBrandon | Cloudflare TSE i wish there was a way to query the api endpoint like

Erisa•10/17/23, 11:24 PM

Thats available in https://api.cloudflare.com/schemas.json but as you can see there are a loooot, for rulesets its best to go through the docs like james linked

Brandon | Cloudflare TSEOP•10/17/23, 11:27 PM

that json is 9.5 MB if you format and save it dogekek

Erisa•10/17/23, 11:35 PM

Yeah

Avi•10/17/23, 11:38 PM

does anyone know of services that will cause cloudflare to automatically try to crawl my pages project? i'm getting some mysterious visits from what seem like bots, and i can't quite figure out where they're from.

Avi•10/17/23, 11:38 PM

theyre not doing anything abusive, just visiting and leaving it seems — but with javascript enabled

MattIPv4•10/17/23, 11:39 PM

Are you sure they're Cloudflare?

MattIPv4•10/17/23, 11:39 PM

Or just bots, 'cause its the internet?

Avi•10/17/23, 11:40 PM

no idea, could be non-cloudflare bots, but i don't know how they would know our website exists. it's not attached to a domiain name, just using the default pages domain name

MattIPv4•10/17/23, 11:40 PM

Certificate transparency logs

Avi•10/17/23, 11:40 PM

i'm not familiar

Avi•10/17/23, 11:40 PM

what's that?

MattIPv4•10/17/23, 11:40 PM

https://crt.sh/?q=pages.dev

Avi•10/17/23, 11:40 PM

ohhh

MattIPv4•10/17/23, 11:41 PM

https://certificate.transparency.dev/

Avi•10/17/23, 11:41 PM

duh, you're right, i guess since each of these has a certificate they're part of the public record

MattIPv4•10/17/23, 11:41 PM

indeed

Avi•10/17/23, 11:42 PM

so, that's totally possible someone's scraping this data. but in order to trigger the behavior i'm seeing, you'd need JS enabled AND a mobile useragent. i guess it's possible for such a bot to exist, but seems pretty unlikely

Avi•10/17/23, 11:42 PM

maybe i need a more colorful imagination

MattIPv4•10/17/23, 11:42 PM

That being said, I thought pages had changed how they issue certs to stop this happening

Avi•10/17/23, 11:43 PM

yeah this list stops at 2021-06-17

MattIPv4•10/17/23, 11:43 PM

But generally speaking, if something is on the internet, something will find it

Avi•10/17/23, 11:43 PM

yeah... i just want to know who. are there logs for traffic to my pages project somewhere?

MattIPv4•10/17/23, 11:44 PM

Someone else here might be more knowledgeable on what exactly enables discovery of pages projects quickly after deployment

MMattIPv4 That being said, I thought pages had changed how they issue certs to stop this h...

Chaika•10/17/23, 11:45 PM

There's a wildcard cert *.pages.dev*.pages.dev which covers the production branch before the project specific one gets issued, and then each Project gets issued a wildcard *.pages-project.pages.dev, pages-project.pages.dev*.pages-project.pages.dev, pages-project.pages.dev for any previews/branches

MattIPv4•10/17/23, 11:45 PM

As for logs, iirc you'll either want to implement your own middleware to do that, or attach a custom domain and use Cloudflare's standard web anayltics

Chaika•10/17/23, 11:46 PM

Can use bulk redirects to redirect people from your pages.dev to your Custom Domain, and then Web Traffic Analytics if Pro or higher to see specific IPs and such

Avi•10/17/23, 11:46 PM

this? https://dash.cloudflare.com/?to=/:account/web-analytics

AAvi this? https://dash.cloudflare.com/?to=/:account/web-analytics

Chaika•10/17/23, 11:48 PM

Pages does have one click support for that, but that won't give you user IPs or anything, just generic country, device/browser/os, paths, hosts, referrars

Avi•10/17/23, 11:48 PM

would it give UAs?

Avi•10/17/23, 11:49 PM

also, our setup is a bit odd — we have a single domain attached to a worker that acts as a transparent reverse proxy across multiple pages projects

Avi•10/17/23, 11:49 PM

every time we push to main we spin up a whole new pages project

MattIPv4•10/17/23, 11:49 PM

I don't think web analytics gives UA details

MattIPv4•10/17/23, 11:50 PM

web traffic analytics will though for a zone that's pro

AAvi would it give UAs?

Chaika•10/17/23, 11:50 PM

It only gives info parsed from the UA like browser/OS, not the raw UA

CChaika It only gives info parsed from the UA like browser/OS, not the raw UA

Avi•10/17/23, 11:52 PM

and this is only aggregate data right?

MattIPv4•10/17/23, 11:53 PM

Both web analytics and web traffic analytics are aggregated

MattIPv4•10/17/23, 11:53 PM

If you want raw logs, you'll either want instant logs (business) or logpush (ent)

MattIPv4•10/17/23, 11:53 PM

Or again, write middleware in a function on your pages project and do logging yourself

AAvi also, our setup is a bit odd — we have a single domain attached to a worker that...

Chaika•10/17/23, 11:54 PM

You could leverage your worker too, to ship logs off somewhere, Analytics Engine or Queues, or just an http endpoint

Avi•10/17/23, 11:58 PM

thanks. in this particular case, the weird thing is that requests seem to be coming to the pages project itself, bypassing the router

Avi•10/17/23, 11:59 PM

our site isn't set up for this, since we expect all traffic to come from the router worker

Avi•10/17/23, 11:59 PM

that's why i'm curious who is making these requests

Chaika•10/18/23, 12:00 AM

ah yea probably bots off CT Logs. You can use a Bulk Redirect rule to redirect any requests coming to the Pages project itself, and Custom domains would still work. Although if your router is fetching the pages.dev directly, would break

Avi•10/18/23, 12:01 AM

right yeah that's exactly what happens. the router transparently reverse proxies the

of the pages.dev directly. the pages.dev is configured to use all absolute URLs for CSS and JS, so all of those are made directly to the pages.dev project, so we don't have to add extra hops through the router

our api docs are a disaster

Similar Threads

Similar Threads

Similar Threads