Cloudflare Developers•3y ago•

105 replies

Hey I think someone is attacking our DOs

Hey I think someone is attacking our DOs:

DanTheGoodmanOP•9/14/23, 6:32 PM

anyone have ideas on how to verify this?

DanTheGoodmanOP•9/14/23, 6:33 PM

we haven't deployed any code for months on DOs

DanTheGoodmanOP•9/14/23, 6:33 PM

or things that connect to DOs IIRC, certainly not in the last ~40 min

DanTheGoodmanOP•9/14/23, 6:35 PM

I can't even tell what DO it's from

DanTheGoodmanOP•9/14/23, 6:40 PM

the log isnt' particularly helpful either...

message.txt2.69KB

DanTheGoodmanOP•9/14/23, 6:41 PM

ther eisn't even an exception in the exception log

DanTheGoodmanOP•9/14/23, 6:41 PM

oh yeah that's weird... had to be an attack

DanTheGoodmanOP•9/14/23, 7:08 PM

happened again in a burst, does not seem to be a isngle object so I might guess it's an infra issue on CF's side?

DanTheGoodmanOP•9/14/23, 7:09 PM

added the ID of the object, not all the same, very high cardinality

DanTheGoodmanOP•9/14/23, 7:10 PM

we are getting hundreds of thousands of these

DanTheGoodmanOP•9/14/23, 7:10 PM

tens per second

DanTheGoodmanOP•9/14/23, 7:11 PM

Milan•9/14/23, 7:15 PM

Can you DM me your account ID, I can take a look to see if something is weird

DanTheGoodmanOP•9/14/23, 7:15 PM

yep

DanTheGoodmanOP•9/14/23, 7:17 PM

dm'd, discord might have suppressed the DM notification @milan_cf

DanTheGoodmanOP•9/14/23, 7:17 PM

had another burst a bit ago too

Milan•9/14/23, 7:18 PM

I got the dm, looking

Milan•9/14/23, 7:28 PM

This started about an hour ago?

Milan•9/14/23, 7:29 PM

18:30 utc or so?

Milan•9/14/23, 7:33 PM

Ah no I see, from 17:50 onwards, definitely see an increase in # invocations

DanTheGoodmanOP•9/14/23, 7:33 PM

Yeah just before then

DanTheGoodmanOP•9/14/23, 7:34 PM

And there’s no reason any individual durable object should be getting even within 20 or 30% of that connection limit

DanTheGoodmanOP•9/14/23, 7:34 PM

Or rather 20% of that limit

DanTheGoodmanOP•9/14/23, 7:34 PM

Largest we know of would be 7k peak

DanTheGoodmanOP•9/14/23, 7:35 PM

Alerts

DanTheGoodmanOP•9/14/23, 7:35 PM

Cloudflare log push to a Grafana stack

DanTheGoodmanOP•9/14/23, 7:42 PM

another one

Milan•9/14/23, 7:45 PM

Yeah, I see a linear increase in the number of websocket connections to some of your DOs

DanTheGoodmanOP•9/14/23, 7:46 PM

how high is the cardinality? Seemed pretty high from the logs when I added the IDs to the urls in the workers

DanTheGoodmanOP•9/14/23, 7:46 PM

(query param)

DanTheGoodmanOP•9/14/23, 7:46 PM

I also checked twitch and none of the 4 people above that viewer count are using our tools

Milan•9/14/23, 8:21 PM

Monitoring of ws hibernation isn't really good enough to know how many DO instances were hitting that limit, I can see there were about 900 instances, most of which received very few messages (under 10), between 17:50utc and 20:00utc (so the last 2 hours)

Milan•9/14/23, 8:25 PM

Bit over 700 of them received 10 or fewer messages

DanTheGoodmanOP•9/14/23, 8:25 PM

that's expected

Milan•9/14/23, 8:26 PM

How many connections are you generally expecting to get to a single DO instance? How are you connecting to the DO?

DanTheGoodmanOP•9/14/23, 8:32 PM

@milan_cf typically <100, but some can be in the thousands. We are connecting through browser websockets

DanTheGoodmanOP•9/14/23, 8:33 PM

ah more

DanTheGoodmanOP•9/14/23, 9:54 PM

@milan_cf any luck? It's pretty constant right now

Milan•9/15/23, 12:03 AM

We don't think the issue is with our infrastructure, there is a significant increase in invocations to your durable object namespace starting at 17:50 UTC, and its been sustained for a while. It's likely that someone is opening a lot of websocket connections to your DOs and forcing you to hit the connection limit.

DanTheGoodmanOP•9/15/23, 12:04 AM

Gotcha, so perhaps an attack then because we pushed no code that modifies how we connect to clients. In my checking of the pretty unhelpful exception logs I do see they are quite spread out among US and EU, but a large number coming from eastern EU. Is there any way to check that? Logpush does not give us that info and this exception happens before our code runs it seems

DanTheGoodmanOP•9/15/23, 12:04 AM

Unless maybe we can get that info in the worker before connecting to the DO?

Milan•9/15/23, 12:09 AM

this exception happens before our code runs it seems

You mean logpush or the DO code?

Milan•9/15/23, 12:10 AM

This exception should be from acceptWebSocket()acceptWebSocket() throwing in the DO

DDanTheGoodman the log isnt' particularly helpful either...

DanTheGoodmanOP•9/15/23, 12:10 AM

see this

DanTheGoodmanOP•9/15/23, 12:10 AM

there's no exception in the exceptions array

DanTheGoodmanOP•9/15/23, 12:11 AM

maybe that's not the right exception, but then what is that exception lol

Milan•9/15/23, 12:12 AM

Where did you get the 32k connection limit exception then?

DanTheGoodmanOP•9/15/23, 12:12 AM

logpush

DanTheGoodmanOP•9/15/23, 12:13 AM

that message from the screenshot is literally all logpush was sending, so I looked into the function call logs for more info

Hey I think someone is attacking our DOs

Similar Threads

Similar Threads

Similar Threads