Is sending to WebSockets from a DO fully blocking?

We have a simple DO that clients connect to via WS and we use hibernation. The DO has an endpoint that our backend can hit to send out a broadcast to all connected clients.

Sometimes the trigger http call times out, even after 5 retries with backoff. My first attempt to fixing this was to change the trigger endpoint to just schedule an alarm, which then handles the sending, but that didn't help.
I assume that is because the DO is single-threaded and so, if the alarm is running, all incoming requests are blocked.

Today I noticed that this happened even though only ~10 connections were open, so the only explanation I have is that one of them is semi-broken, or just very slow, and that the send() call is fully blocking.

Is that the case? If so, why? This seems to a unnecessarily strict, as sending a WS message could just put it into an outbound queue.

Martin 0x522EOP•10/24/23, 12:18 PM

@Helpflare who can help with this?

Milan•10/26/23, 1:27 PM

Sending a websocket message does put it into a queue, it doesn't block.

Milan•10/26/23, 1:29 PM

Your backend sends a broadcast request to your DO (the broadcast endpoint), and then you do what exactly? Get back all your websockets and send them each a message? wdym by "http call times out"? Are you able to confirm that any of the ws get the broadcasted message?

Martin 0x522EOP•10/26/23, 1:32 PM

Yes exactly, the endpoint just forwards the fetch to the DO and the DO then iterates over all the websockets and sends each the same message.
And that fetch then times out while waiting for the response - which would just be an empty 200.
So all of this should be basically instant, so I don't understand why we sometimes get timeouts

Martin 0x522EOP•10/26/23, 1:32 PM

As I can't reproduce it, I also don't know of some messages get out

Martin 0x522EOP•10/26/23, 1:34 PM

The only time I could reproduce it, I had just closed the websocket client, so I don't know if it WOULD have received the message.
But as this case triggered the issue, I assumed it might either be blocking or at least that the fetch only returns when the DO is done with all WS messages

Martin 0x522EOP•10/26/23, 1:35 PM

Also the send() message sometimes throws an exception that the WS is already closed. How does that work if that method only gets queued @milan

Milan•10/26/23, 1:42 PM

I'll check with the team but I'm fairly certain the response wouldn't be blocked by outbound messages waiting in the queue, that should just prevent the DO from being evicted.

Milan•10/26/23, 1:46 PM

If you get an exception from send()send() that the WS was already closed that means ws.close()ws.close() was called previously. IIRC it's possible that you called ws.close()ws.close() and the client hasn't responded with its own close message, so we wouldn't remove it from the list of hibernatable websockets. In that case, you could get it back via getWebSockets()getWebSockets() even though you can't send outbound messages anymore.

Martin 0x522EOP•10/26/23, 2:31 PM

Thanks, yeah the latter makes sense. Just the timeouts are a mystery to me and I don't know how else to debug

Milan•10/26/23, 2:57 PM

I assume you're

await

await

ing the fetch to the DO and that is what's timing out? Maybe wrap your ws.send()ws.send() in a try...catchtry...catch? Do you have a webSocketClose()webSocketClose() handler that closes the DO's websocket?

Martin 0x522EOP•10/26/23, 3:02 PM

We are try catching and then just ignoring that error. We do have that handler, but we don't do anything with them

Martin 0x522EOP•10/26/23, 3:05 PM

This is the code. Had to rename from .ts

WebsocketManager.tscode3.81KB

index.tscode1.28KB

Milan•10/26/23, 4:20 PM

You're not timing out when you call broadcast via alarm()alarm()?

Milan•10/26/23, 4:29 PM

I don't know that we can do much without a repro here, you should be able to return a response before all messages are sent so send()send() wouldn't block. Something else must be happening, maybe it was caused by additional work running in the constructor?

Is sending to WebSockets from a DO fully blocking?

Similar Threads

Similar Threads

Similar Threads