I believe they were asking about whether time advances when a setInterval callback is triggered (tim

I believe they were asking about whether time advances when a setInterval callback is triggered (time is locked except for I/O in Workers)

Soma•9/14/25, 1:56 PM

oh, sorry i misunderstood

IIsaac McFadyen I believe they were asking about whether time advances when a setInterval callba...

Avi•9/14/25, 2:51 PM

yes that’s correct

Avi•9/14/25, 2:52 PM

recently someone discovered an exploit where they could cause our DO’s time to “slow down” by flooding it with websocket messages

Avi•9/14/25, 2:53 PM

normally time is set to

Date.now()

Date.now()

every 1000 ms via

setInterval()

setInterval()

Avi•9/14/25, 2:54 PM

and i’m trying to figure out how that’s possible

AAvi normally time is set to `Date.now()` every 1000 ms via `setInterval()`

Larry•9/14/25, 3:55 PM

What exactly do you mean? Have you observed that this does indeed update

Date.now()

Date.now()

LLarry What exactly do you mean? Have you observed that this does indeed update `Date.n...

Avi•9/14/25, 5:03 PM

i’m asking actually the reverse almost - i’m finding that it is NOT updating date.now()

Avi•9/14/25, 5:04 PM

one of our players found that by spamming messages to our gameserver DO, they could “slow time” on the server

Avi•9/14/25, 5:04 PM

on the server, the time is set by updating an in-memory variable to the result of Date.now() every 1000ms

Avi•9/14/25, 5:05 PM

i’m trying to figure out how exactly sending a lot of websocket messages is causing that clock skew

Avi•9/14/25, 5:05 PM

apparently they managed to get it to drift by up to 2 hours once

Avi•9/14/25, 5:06 PM

my working assumption is that when there’s a lot of incoming websocket messages queued, they’ll all get handled synchronously without updating Date.now()

Avi•9/14/25, 5:08 PM

i guess somehow the other possibility is that the javascript internal tick loop is just so behind it’s still executing code with older values of Date.now()

Avi•9/14/25, 5:08 PM

but that seems even stranger

Avi•9/14/25, 5:09 PM

this player managed to demonstrate that after “winding up the clock” (spamming a DO), if they terminate the connection (presumably clearly the queue) they can essentially cause time to “wind forward” very quickly

Avi•9/14/25, 5:09 PM

as if the setIntervals that were meant to run every 1000ms are running every 50ms

Avi•9/14/25, 5:09 PM

i wouldn’t have believed it if i didn’t see it with my own eyes

Isaac McFadyenOP•9/14/25, 5:17 PM

How fast are they spamming messages? I wonder if perhaps the event loop is too busy processing messages to execute all of the setIntervals properly, so it queues them, and then when they disconnect it quickly runs them all?

Isaac McFadyenOP•9/14/25, 5:18 PM

Workers are single-threaded, are you doing any async work in the WS handler?

Larry•9/14/25, 10:58 PM

You may be more interested in understanding how it behaves than fixing it. I know I am. But a fix for your time manipulators, could be... in your message/fetch handlers, you took the max of Date.now() and your manually maintained clock, then set your manually maintained clock to that value and reset the 1000ms timer. Then it will never get more than a few milliseconds off.

Avi•9/15/25, 12:37 AM

good questions @Isaac | AS213339! any idea how i could get some visibility into "how busy the event loop is"? what you describe about the setIntervals being queued to run in the event loop seems plausible to me - that certainly seems consistent with the timer "spinning fast" after they leave

Avi•9/15/25, 12:38 AM

some async does happen in the WS handler, but most of the async work happens in a separate

setInterval()

setInterval()

. changes are made in memory and periodically synced to the database

Isaac McFadyenOP•9/15/25, 12:38 AM

I don't think there's much visibility into the event loop in Workers sadly. Larry's suggestion above might be a good thing to try though.

Isaac McFadyenOP•9/15/25, 12:39 AM

It might come down to runtime fairness and whether requests will ever be held for the (short) time it takes to run the queued setInterval.

Avi•9/15/25, 12:41 AM

i'm not sure i understand @Larry's suggestion. by "manually maintained clock", does that refer to the in-memory variable I'm storing the value of

Date.now()

Date.now()

?

essentially my code is like:

setInterval(() => {
  this.currentTime = Date.now();
  tickGame();
}, 1000);

setInterval(() => {
  this.currentTime = Date.now();
  tickGame();
}, 1000);

Avi•9/15/25, 12:42 AM

i guess the biggest question is: is the clock skewing due to Date.now() returning 'stale' data due to I/O starvation OR is it because the callbacks are getting 'queued' waiting to be run by the JS event loop and are then all running very rapidly

Avi•9/15/25, 12:43 AM

in other words, is it that

Date.now()

Date.now()

is stale? or are the callbacks just getting executed less frequently than expected?

Isaac McFadyenOP•9/15/25, 1:01 AM

So what I'm guessing is the case here, although I have no basis to back it up since I haven't looked at the runtime code, is that

workerd

workerd

is queuing the setInterval() callbacks. At the moment the interval is added to the queue, it's using that frozen

Date.now()

Date.now()

time for that microtask, but then they sit in the queue a while. So then once the queue is unblocked, they all run very quickly and you see time as though it is "fast-forwarding"

Isaac McFadyenOP•9/15/25, 1:03 AM

Someone on the runtime team would have to confirm this though. I'm not sure how workerd handles the Date.now() freezing and if it pulls the last value at the time of the queued microtask or at the time of actually calling Date.now()

Avi•9/15/25, 1:04 AM

@Isaac | AS213339 this makes sense - and IF the callbacks are queued and then running rapidly, it stands to reason they might only advance their timer after each one, as some I/O happens

Avi•9/15/25, 1:04 AM

the shocking thing is that the "fast-forwarding' is very smooth

Avi•9/15/25, 1:05 AM

Avi•9/15/25, 1:05 AM

this is a video recording from the game showing the fast-forwarding exploit in action

Isaac McFadyenOP•9/15/25, 1:05 AM

Uhhh that's very odd.

Isaac McFadyenOP•9/15/25, 1:07 AM

One workaround you could do though is change from a setInterval to a setTimeout and use delta time. Not sure if this would work for your use-case though?

let lastTicked = Date.now();
const gameLoop = () => {
   const delta = Date.now() - lastTicked; // the *actual* number of ms since last tick
   this.currentTime += delta;
   doGameLogic();
   setTimeout(gameLoop, 1000);
};
gameLoop();

let lastTicked = Date.now();
const gameLoop = () => {
   const delta = Date.now() - lastTicked; // the *actual* number of ms since last tick
   this.currentTime += delta;
   doGameLogic();
   setTimeout(gameLoop, 1000);
};
gameLoop();

Avi•9/15/25, 1:08 AM

that would require a lot of re-engineering of the server code, but that approach is quite common in game engines more generally

Avi•9/15/25, 1:09 AM

certainly that's how e.g. rendering engines do it

Isaac McFadyenOP•9/15/25, 1:09 AM

Yeah that's what I was basing this off

Isaac McFadyenOP•9/15/25, 1:09 AM

Not sure what alternatives would be here though, unfortunately... if it is indeed some sort of queue then you wouldn't have much control over it user-side.

Avi•9/15/25, 1:11 AM

well, one crazy idea i had was to introduce a 'time keeper' service/API that would return a truly global source of truth

Avi•9/15/25, 1:12 AM

that doesn't really solve things if the issue is setIntervals building up though, but it would at least be a trustworthy Date.now() source

Avi•9/15/25, 1:12 AM

come to think of it, the behavior i showed with the timer spinning like that has to mean the issue is queueing of the callbacks

Avi•9/15/25, 1:13 AM

otherwise it wouldn't update so quickly

Avi•9/15/25, 1:13 AM

or, i guess another possibility is that the queue is is processed very quickly, but the outgoing websocket messages are queued

Isaac McFadyenOP•9/15/25, 1:13 AM

Yeah, that's what it seems like. It's like the runtime suddenly has time and executes all of the queued tasks really quickly so it seems like it's fast-forwarding.

Isaac McFadyenOP•9/15/25, 1:14 AM

Oh that might be it as well... how easy would it be to add a timestamp to the WS message of "sent at this time"?

Isaac McFadyenOP•9/15/25, 1:14 AM

Then you can see when the Worker thinks it sent the message

Avi•9/15/25, 1:14 AM

not impossible but also can we really trust that timestamp? xD

Isaac McFadyenOP•9/15/25, 1:14 AM

Yeaaaah :NotLikeThis:

Avi•9/15/25, 1:14 AM

since Date.now() is itself perhaps unreliable

let lastTicked = Date.now(); const gameLoop = () => { const delta = Date.now() - lastTicked; // the *actual* number of ms since last tick this.currentTime += delta; doGameLogic(); setTimeout(gameLoop, 1000); }; gameLoop();