Do you know anything about scraping on workers? I assume a lot of the CF zones are IP banned for abu
Do you know anything about scraping on workers? I assume a lot of the CF zones are IP banned for abuse
MNope. I have used Google Cloud Functions for scraping tho and that works just about fine. At least on the APIs I was scraping
MBut I assume if it's blocking CF Workers, then it's also blocking others
DYeh haven’t tested in prod yet so not sure once this thing scales up
DJust gonna cache as much as I can into KV I guess
MIf you aren't doing anything malicious you can always add some "hello-its-me: your-friend-scraper" header to requests and tell to API / site you scrape to plz not block that
MWin-win situation. They can keep blocking malicious bots and you can get the needed info
CWorkers for non cf websites use https://www.cloudflare.com/ips/ IPs, same as with normal proxy.
They're def not meant for doing anything unauthorized, all outbound requests contain the CF-Worker header with the name of your zone (https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#cf-worker), Discord for example blocks all Workers from their CDN except for temp. media and avatars
They're def not meant for doing anything unauthorized, all outbound requests contain the CF-Worker header with the name of your zone (https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#cf-worker), Discord for example blocks all Workers from their CDN except for temp. media and avatars
DI wish lol, all the sites I’m scraping have TOS strictly forbidding scraping
DDamn
DDo I just go with AWS then? Or do you know of or recommend any Cloudflare-like/serverless services that don’t have this bottleneck
MAWS is gonna be quite the same. AWS adds pile of
X-Amzn-* header for outgoing requests IIRCMAnd they are all still part of AWS IP range
DI thought Lambda didn’t have fixed IPs
MDAlright I’ll do some research thanks
M
AQuick question
From my understanding, calls to service bindings uses the same memory space and thread as the calling worker.
So does calling that service binding count as an invocation against that worker?
Basically what im asking, if I have a worker that always calls a service binding for another worker, and that first worker gets 1 million requests, will I be billed for 1 million reuqests or 2 million requests?
From my understanding, calls to service bindings uses the same memory space and thread as the calling worker.
So does calling that service binding count as an invocation against that worker?
Basically what im asking, if I have a worker that always calls a service binding for another worker, and that first worker gets 1 million requests, will I be billed for 1 million reuqests or 2 million requests?
K2
A:/ ok thanks.
seems a bit excessive to bill that as another invocation when it has to work within the limits of the first invocation IMO
seems a bit excessive to bill that as another invocation when it has to work within the limits of the first invocation IMO
AOn the new standard pricing, you pay for the CPU time anyway, so you aren't charged for the number of requests, so it's not really going to impact your billing
KYou still pay for requests.
AAh yes sorry it's a combo of them both isn't it
KYep.
Bundled = reqs only.
Unbound = reqs + wall time.
Standard (new) = reqs + CPU time.
Bundled = reqs only.
Unbound = reqs + wall time.
Standard (new) = reqs + CPU time.
QAnyone knows if we can disable
Tried deleting the route handlers through the dashboard, but they keep coming back after deploying with wrangler cli
xxxxx.workers.dev default routes from wrangler.toml? I'm using sentry integration beta and noticing a lot of no fetch handler defined errors, probably mostly bots trying to ping the host, but I don't need any routes for the queue. Tried deleting the route handlers through the dashboard, but they keep coming back after deploying with wrangler cli
Cworkers_dev = falseCiirc having no routes/custom domains at all breaks Quick edit though, or used to. Although Queues wouldn't work with that anyway
Qthanks for the solution and the hint! @Chaika np when it breaks quick edit
MHello ,
I have cloudflare worker , and I want to use WAF custom rule , in the worker
so for example
if (( waf custom rule = example)
do something
how I can do that with cloudflare workers
thanks
I have cloudflare worker , and I want to use WAF custom rule , in the worker
so for example
if (( waf custom rule = example)
do something
how I can do that with cloudflare workers
thanks
KYou'd need to wait for Snippets if you want to run a Worker in response to a WAF rule.
Mwhat you mean by Snippets , can you explain in more details please
KCloudflare Snippets (alpha) provide a flexible way to customize the behavior of your website or application using short pieces of JavaScript code. Use …
Mits still beta and I dont have access to it , is there any other way ??
KNope, Workers currently are always invoked on a given route.
KWhat are you trying to do, invoke a Worker based on a rule or replicate rule logic in a Worker?
Minvoke a Worker based on a WAF rule , (( so I just want the Worker to be triggered only when specific CUSTOM waf rule triggered )) currently its triggered on all (( skip )) waf rules
KYeah, you can only do that with Snippets when they become available.
UWhy do you want to trigger a worker by a WAF rule though? 
Mvpn detection system , currently there are a lot of good bots being detected as vpn , and they got blocked by the worker
Mand I want the worker to not work on (( custom waf rule == known bots ))
KJust to make sure, you are aware that Known Bots means it's a good bot.
Myes I know
KGotcha
Mcurrently my vpn detection system see that most of the good bots as proxy and vpn and they are blocking them
Mso I want to bypass worker if it hit the known bots rule
Mwow ! thats great , by the way what you mean by (( iirc that' the name ))
KThere's
verifiedBotCategory which will be "" if it's not a known bot, otherwise containing the category of the bot.Mwow thank you !
Mif (request.cf.verifiedBot) {
// Logic for handling verified bots
// You can choose to bypass certain operations or handle them differently
return fetch(request);
}so it should be something like that , verified and known bots will hit this
// Logic for handling verified bots
// You can choose to bypass certain operations or handle them differently
return fetch(request);
}so it should be something like that , verified and known bots will hit this
SOn the "Overview" page for Workers & Pages, I can see the sum of all my requests and CPU time, etc.
It is not possible, however, to get a breakdown on which Pages projects are using how much unless I click into every project, go to function metrics, and then adjust the timeframe to something other than "last 24 hours"
Is there any way I can view a usage / billing breakdown, even via the API?
It is not possible, however, to get a breakdown on which Pages projects are using how much unless I click into every project, go to function metrics, and then adjust the timeframe to something other than "last 24 hours"
Is there any way I can view a usage / billing breakdown, even via the API?
DIs there an easy way to throttling my subrequests globally?
