Yes, reached out. They mentioned they'd get this merged in tomorrow or Monday.
Yes, reached out. They mentioned they'd get this merged in tomorrow or Monday.
AHi, I'm using the Agents SDK for DO's, but I'm not clear on the right way to terminate a DO instance. I've tried doing something like and the destroy() method just does a but this causes a bunch of errors to show up. Based on the errors, it looks like the DO is trying to do some schedule related things in its storage, which is now gone.
Am I doing something wrong or is there a correct way to permanently terminate an Agent DO?
Am I doing something wrong or is there a correct way to permanently terminate an Agent DO?
L is higher-level layer provided by the Agent base class that uses
setAlarm
AMakes sense, thank you!
JNice! Thats great!
Adoes a setInterval callback “count” as I/O for the purpose of causing Date.now() to advance?
Sif you use setInterval the DO's runtime is kept active as per the lifecycle docs
II believe they were asking about whether time advances when a setInterval callback is triggered (time is locked except for I/O in Workers)
Soh, sorry i misunderstood
Ayes that’s correct
Arecently someone discovered an exploit where they could cause our DO’s time to “slow down” by flooding it with websocket messages
Anormally time is set to
Date.now()and i’m trying to figure out how that’s possible
LWhat exactly do you mean? Have you observed that this does indeed update
Date.now()i’m asking actually the reverse almost - i’m finding that it is NOT updating date.now()
Aone of our players found that by spamming messages to our gameserver DO, they could “slow time” on the server
Aon the server, the time is set by updating an in-memory variable to the result of Date.now() every 1000ms
Ai’m trying to figure out how exactly sending a lot of websocket messages is causing that clock skew
Aapparently they managed to get it to drift by up to 2 hours once
Amy working assumption is that when there’s a lot of incoming websocket messages queued, they’ll all get handled synchronously without updating Date.now()
Ai guess somehow the other possibility is that the javascript internal tick loop is just so behind it’s still executing code with older values of Date.now()
Abut that seems even stranger
Athis player managed to demonstrate that after “winding up the clock” (spamming a DO), if they terminate the connection (presumably clearly the queue) they can essentially cause time to “wind forward” very quickly
Aas if the setIntervals that were meant to run every 1000ms are running every 50ms
Ai wouldn’t have believed it if i didn’t see it with my own eyes
IHow fast are they spamming messages? I wonder if perhaps the event loop is too busy processing messages to execute all of the setIntervals properly, so it queues them, and then when they disconnect it quickly runs them all?
IWorkers are single-threaded, are you doing any async work in the WS handler?
LYou may be more interested in understanding how it behaves than fixing it. I know I am. But a fix for your time manipulators, could be... in your message/fetch handlers, you took the max of Date.now() and your manually maintained clock, then set your manually maintained clock to that value and reset the 1000ms timer. Then it will never get more than a few milliseconds off.
Agood questions @Isaac | AS213339! any idea how i could get some visibility into "how busy the event loop is"? what you describe about the setIntervals being queued to run in the event loop seems plausible to me - that certainly seems consistent with the timer "spinning fast" after they leave
Asome async does happen in the WS handler, but most of the async work happens in a separate . changes are made in memory and periodically synced to the database
II don't think there's much visibility into the event loop in Workers sadly. Larry's suggestion above might be a good thing to try though.
IIt might come down to runtime fairness and whether requests will ever be held for the (short) time it takes to run the queued setInterval.
Ai'm not sure i understand @Larry's suggestion. by "manually maintained clock", does that refer to the in-memory variable I'm storing the value of
essentially my code is like:
Date.now()essentially my code is like:
Ai guess the biggest question is: is the clock skewing due to Date.now() returning 'stale' data due to I/O starvation OR is it because the callbacks are getting 'queued' waiting to be run by the JS event loop and are then all running very rapidly
Ain other words, is it that
Date.now()So what I'm guessing is the case here, although I have no basis to back it up since I haven't looked at the runtime code, is that
workerdDate.now()Someone on the runtime team would have to confirm this though. I'm not sure how workerd handles the Date.now() freezing and if it pulls the last value at the time of the queued microtask or at the time of actually calling Date.now()
A@Isaac | AS213339 this makes sense - and IF the callbacks are queued and then running rapidly, it stands to reason they might only advance their timer after each one, as some I/O happens
Athe shocking thing is that the "fast-forwarding' is very smooth
AAthis is a video recording from the game showing the fast-forwarding exploit in action
IUhhh that's very odd.
IOne workaround you could do though is change from a setInterval to a setTimeout and use delta time. Not sure if this would work for your use-case though?
Athat would require a lot of re-engineering of the server code, but that approach is quite common in game engines more generally
Acertainly that's how e.g. rendering engines do it
IYeah that's what I was basing this off 
INot sure what alternatives would be here though, unfortunately... if it is indeed some sort of queue then you wouldn't have much control over it user-side.
Awell, one crazy idea i had was to introduce a 'time keeper' service/API that would return a truly global source of truth
Athat doesn't really solve things if the issue is setIntervals building up though, but it would at least be a trustworthy Date.now() source
Acome to think of it, the behavior i showed with the timer spinning like that has to mean the issue is queueing of the callbacks