K6 load testing and 429s

Hey team

Currently been working on load testing a simple worker (just a JS 10 liner kind of deal) we have (non-free/ent tier) and have found that when trying to perform load testing using K6 we begin to see a flood of 429's

I did find this post: https://community.cloudflare.com/t/can-requests-to-workers-get-rate-limited/435598/2

Which suggests i'm encountering

on paid it defends against a small amount of IPs sending too many requests

on paid it defends against a small amount of IPs sending too many requests

on paid it defends against a small amount of IPs sending too many requests

on paid it defends against a small amount of IPs sending too many requests

But I cannot seem to find a ways to configure either whitelisting the IP's i'm using or to temporarily disable this throttling feature.

Where can I find more information about this? Is there configuration I can see either on the worker level or in the portal around this?

Erisa•6/20/24, 10:09 PM

https://developers.cloudflare.com/workers/platform/limits/#request

Cloudflare’s abuse protection methods do not affect well-intentioned traffic. However, if you send many thousands of requests per second from a small number of client IP addresses, you can inadvertently trigger Cloudflare’s abuse protection. If you expect to receive 1015 errors in response to traffic or expect your application to incur these errors, contact Cloudflare support to increase your limit. Cloudflare’s anti-abuse Workers Rate Limiting does not apply to Enterprise customers.

CodexOP•6/20/24, 10:16 PM

Thanks for that, so I've read that and while it talks to a lot of the implicit limitations on free tiers and makes mention of handling 429's if you're programatically hitting the worker like:

However if you are calling your Worker programmatically, you can detect the rate limit page and handle it yourself by looking for HTTP status code 429.

However if you are calling your Worker programmatically, you can detect the rate limit page and handle it yourself by looking for HTTP status code 429.

I'm still failing to see how I can outright disable this protection measure or whitelist specific IPs while this testing is being performed

Erisa•6/20/24, 10:19 PM

There's a limit that applies to paid plans as well, and the solution is to contact support:

If you expect to receive 1015 errors in response to traffic or expect your application to incur these errors, contact Cloudflare support to increase your limit.

CodexOP•6/20/24, 10:20 PM

I was reading through this documentation too: https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/#rate-limiting

But Rate limits that are applied after your Worker startsRate limits that are applied after your Worker starts suggests this isn't where I want to be lol

EErisa There's a limit that applies to paid plans as well, and the solution is to conta...

CodexOP•6/20/24, 10:20 PM

Ah awesome, will do, thank you

CCodex I was reading through this documentation too: <https://developers.cloudflare.com...

Erisa•6/20/24, 10:20 PM

That's completely different

CodexOP•6/20/24, 10:21 PM

yeah that's where I got to, just clarifying what I have read and done to this point :)

Erisa•6/20/24, 10:21 PM

By the way if you want to verify that the limit you're hitting is indeed the anti-abuse limit, trigger it on a custom domain (not workers.dev) and then check Security > Events on the dash. If you see events with a rule ID of workerworker then thats definitely it

Erisa•6/20/24, 10:22 PM

When you contact support you'll be asked what zones(domains) you need it disabled on, and you can share either the workers.dev subdomain or your custom domain name(s)

CodexOP•6/20/24, 10:23 PM

Great to know, thank you!

CodexOP•6/20/24, 10:35 PM

So interestingly, it appears that we get the 429 as a result of the subrequest opposed to the top level request. Security > EventsSecurity > Events

The services (Azure Functions) being hit by the worker work perfectly when hit directly, but we start seeing the 429s when the worker is involved (CF Worker -> Azure Functions). These subrequests are where we're observing 10151015s

Security > EventsSecurity > Events doesn't appear to have any security events matching that description, so something else perhaps.

Regardless though, we'll put something through to CF support and go from there.

CCodex So interestingly, it appears that we get the 429 as a result of the subrequest o...

Erisa•6/21/24, 1:31 AM

Do you actually see 1015 in the resposne body?

K6 load testing and 429s

Similar Threads

Similar Threads

Similar Threads