22 replies

Would it be possible to have the runtime

Would it be possible to have the runtime store only the latest N WebSockets, and then GC the rest?

Milan•1/29/24, 5:37 PM

This might be possible, though I don't know enough about if we can force JS to GC an object that still has references (I suspect no and also we shouldn't do that). Ex. Req1Req1 broadcasts to all currently connected websockets (got them via getWebSockets()getWebSockets()), then does an await and yields event loop. Req2...ReqNReq2...ReqN create new websockets and those get used a bit, so we evict all WS prior to Req2Req2. When Req1Req1 is done its IO and is scheduled to run again, when it tries to refer to its array of websockets it got from getWebSockets()getWebSockets(), something weird would happen

Milan•1/29/24, 5:39 PM

Would also have to consider different eviction policies (and if that should be configurable by developers)

MMilan This might be possible, though I don't know enough about if we can force JS to G...

Hello, I’m Allie!OP•1/29/24, 5:39 PM

Wait, so if I getWebSockets(), use the WebSocket once, then move out of scope, is the WebSocket Object then garbage collected, or is it stored somewhere else too?

HHello, I’m Allie!Wait, so if I getWebSockets(), use the WebSocket once, then move out of scope, i...

Milan•1/29/24, 5:41 PM

We have another reference to the JS WebSocket elsewhere

MMilan We have another reference to the JS WebSocket elsewhere

Hello, I’m Allie!OP•1/29/24, 5:42 PM

Can you track the active references within user code, then evict if they are only referenced in the backend?

Hello, I’m Allie!OP•1/29/24, 5:44 PM

I assume the backend can recover more easily from no longer having an active object reference

Milan•1/29/24, 5:54 PM

It's a good question, I was thinking the same (a lot of the C++ we wrote for hibernation does ref counting -- specifically knowing if we are holding the only remaining strong ref to some object). I'm not sure we have a way to know that for our JS types today, briefly skimming it looks like we don't. That might be something we could add, but between modifying refcounting + having some type of eviction promise/loop it would be a pretty significant project and would make the websocket/hibernation code (which is already complicated) hard to maintain.

We probably need to refactor the code + finish outgoing hibernation before considering something like this. Tinkering with object lifetimes is one of the more risky things we do on the runtime.

MMilan It's a good question, I was thinking the same (a lot of the C++ we wrote for hib...

Hello, I’m Allie!OP•1/29/24, 5:56 PM

Inversely, what about a function on statestate that signals to the runtime that it is ok for the DO to be restarted? It would at least solve the growing number of objects problem, and might be useful for user code to be able to "clean up" if it generates a lot of in-memory state

Milan•1/29/24, 5:56 PM

Or even delete them when they are added, and only recreate them for the time they need to be processed?

Typically folks will want to do some stuff with the object immediately after accepting it, so it's not clear at what point it would make sense to do this. I suspect if anything what you suggested before (evicting on the backend if there's no user references) makes most sense

MMilan > Or even delete them when they are added, and only recreate them for the time t...

Hello, I’m Allie!OP•1/29/24, 5:57 PM

Yeah sorry, I was kind of thinking about it the wrong way. There, I meant delete them after they are added(but no longer have references in user space).

HHello, I’m Allie!Inversely, what about a function on `state` that signals to the runtime that it ...

Hello, I’m Allie!OP•1/29/24, 5:58 PM

It would also allow update-related restarts to occur at a time most useful for the program. It wouldn't entirely prevent forced-evictions, but it could help some, no?

Milan•1/29/24, 5:58 PM

signals to the runtime that it is ok for the DO to be restarted

As in, signals that after this current request is finished you can immediately evict the instance for hibernation?

MMilan > signals to the runtime that it is ok for the DO to be restarted As in, signal...

Hello, I’m Allie!OP•1/29/24, 6:00 PM

~~That, or a "I've reached a state that I don't think I can recover from, so evict the DO now"~~

Hello, I’m Allie!OP•1/29/24, 6:01 PM

~~Maybe a toggle inside, as a boolean, or something~~

Hello, I’m Allie!OP•1/29/24, 6:02 PM

Actually, never mind. Eviction at end of current requests would probably make more sense

Hello, I’m Allie!OP•1/29/24, 6:03 PM

Also so that another request in progress doesn't get cut off

MMilan > signals to the runtime that it is ok for the DO to be restarted As in, signal...

Hello, I’m Allie!OP•1/29/24, 6:06 PM

Something like

// state.evict stops further events from entering the queue, but
// does not prevent current events from being processed.
this.state.evict(() => {
  // This function is run after all events have completed, but before the DO is evicted.
  // Could be used for final cleanup tasks
});
return new Response(null);

// state.evict stops further events from entering the queue, but
// does not prevent current events from being processed.
this.state.evict(() => {
  // This function is run after all events have completed, but before the DO is evicted.
  // Could be used for final cleanup tasks
});
return new Response(null);

Milan•1/29/24, 6:07 PM

That's come up before but my understanding is there are enough pitfalls that we have decided to punt it indefinitely

MMilan That's come up before but my understanding is there are enough pitfalls that we ...

Hello, I’m Allie!OP•1/29/24, 6:07 PM

Just curious, if you are able to speak about what issues might occur?

Milan•1/29/24, 6:18 PM

Mostly just issues around deterministic behavior. It would be misleading to suggest that this runs when your DO is evicted because eviction could mean anything from: you breached a runtime limit to the machine no longer exists. It would really only be useful in the context of a shutdown due to inactivity. Not saying this isn't useful (I like destructors!) but it's fairly limited relative to what people would expect, and even in the clean shutdown case we have to consider things like
- how long it can run for
- can it do IO
- should we cancel shutdowns if new requests come in (does that mean your shutdown procedure has to be a transaction)

MMilan Mostly just issues around deterministic behavior. It would be misleading to sugg...

Hello, I’m Allie!OP•1/29/24, 6:21 PM

That all makes sense

Milan•1/29/24, 6:27 PM

To summarize, I'll open a ticket internally for evicting inactive hibernatable websockets and write some thoughts so it doesn't get lost here. If you're interested, feel free to open a discussion on the Workerd repo too

Hello, I’m Allie!OP•1/29/24, 7:36 PM

https://github.com/cloudflare/workerd/discussions/1590

GitHub

Hibernation API Memory Management · cloudflare workerd · Discussion...

It was pointed out to me today by @MellowYarker that using the Hibernatable WebSocket API does not preclude running out of memory due to WebSocket objects not being automatically deleted when no lo...

Would it be possible to have the runtime

Similar Threads

Similar Threads

Similar Threads