i did a workaround, but its very ugly! i thought drizzle is popular ORM to use with CloudFlare durab

i did a workaround, but its very ugly!
i thought drizzle is popular ORM to use with CloudFlare durableObject.

message.txt2.78KB

TTuna i did a workaround, but its very ugly! i thought drizzle is popular ORM to use w...

plgingras88•5/20/25, 11:58 AM

Sqlite storage in DO is something very new, it barely ended its beta phasis few weeks ago. Drizzle adopted it fairly quickly, only few weeks ago the typing for DO sqlite wasn't even supported with the orm and now it is, so I feel they are working as fast as they can to adapt accordingly. Give them a little bit of time, DO's a special creature that needs the libs to work a little differently. I wouldn't be surprised your problem may be addressed in a forseable future with user adoption and bug reports.

João Castro•5/20/25, 12:28 PM

Their DO driver has some issues, and unfortunately they don't seem to be paying much attention to it

João Castro•5/20/25, 12:29 PM

I have a PR open since December to fix a bug on it and its still sitting there https://github.com/drizzle-team/drizzle-orm/pull/3746

GitHub

fix durable sqlite transaction return value by joaocstro · Pull Re...

Returns the correct value from Durable Objects SQLite transactions, which were always returning an empty object regardless of the value returned by the callback.

TTuna i did a workaround, but its very ugly! i thought drizzle is popular ORM to use w...

James•5/20/25, 2:32 PM

I use drizzle purely to generate migrations for DO. Then I reuse the drizzle schema at runtime for type checking post-query responses. All actual DO request logic is pure SQL. Has worked pretty well for nearly a year now.

Flux•5/21/25, 9:46 AM

If there are any people here interested in sync engines and building a library around durable objects feel free to send me a message. So far ive taken inspiration from the amazing Zero team: https://zero.rocicorp.dev

Zero Docs

The official documentation for Zero by Rocicorp.

Super Morris•5/21/25, 1:47 PM

Is there a way in DO to wait for a response longer than 30seconds after client disconnected?
Modern API Request take longer than 30 seconds and I need to make sure that the response is processed.
(without queue)

SSuper Morris Is there a way in DO to wait for a response longer than 30seconds after client d...

lambrospetrou•5/21/25, 1:51 PM

You can use Alarms to keep the DO alive, or simply doing

setTimeout

setTimeout

in a few seconds in the future each time. If there is no incoming request or alarm trigger for 10s the DO will be evicted from memory and destroyed after 1-2 minutes, so make sure to schedule one of the above every [5, 10) seconds.

Super Morris•5/21/25, 1:52 PM

so you say, as long as I keep it alive with an alarm it is able to handle the response of an http request?

SSuper Morris Is there a way in DO to wait for a response longer than 30seconds after client d...

Larry•5/21/25, 4:01 PM

Besides alarms, another strategy is to use WebSockets. Request comes in over WebSockets. Optionally you confirm receipt of the request over the WebSocket. However, you push the request to a Worker (with a longer stay-alive time and cpu time billing instead of DOs clock time billing) that accomplishes the request. When it's done, it calls back to the DO that wakes up the hiberated WebSocket and sends the response.

workerBee•5/21/25, 4:13 PM

This API endpoint seems to be busted

https://api.cloudflare.com/client/v4/accounts/:account_id/workers/durable_objects/namespaces/:namespace/objects

https://api.cloudflare.com/client/v4/accounts/:account_id/workers/durable_objects/namespaces/:namespace/objects

WworkerBee This API endpoint seems to be busted `https://api.cloudflare.com/client/v4/accou...

lambrospetrou•5/21/25, 4:33 PM

I reported it to the team and they will take a look. Thanks for reporting.

Marak•5/21/25, 8:58 PM

Something seems very wrong with our Durable Objects today; we are seeing 10,000,000+ ms wall times on requests that take < 1ms CPU. Concerned this may lead to a catastrophic billing situation.

Screenshot_2025-05-21_at_10.55.46_AM.png

Marak•5/21/25, 9:21 PM

^^^ Can anyone recommend the best way to query the current Durable Objects Compute Duration usage for an account? Trying to get a handle on this situation.

MMarak ^^^ Can anyone recommend the best way to query the current Durable Objects Compu...

Chaika•5/21/25, 9:44 PM

Under Compute (Workers) -> Durable Objects, you can see your account usage(request, duration, storage usage, etc)

CChaika Under Compute (Workers) -> Durable Objects, you can see your account usage(reque...

Marak•5/21/25, 9:55 PM

Thank you, that helps. Do you have any idea why the wall time is so high for our Durable Objects? The affected instances shouldn't have any other connected clients or persistent connections. We are performing an RPC from another service to update state inside the DO. CPU time is minimal, but the wall time is holding for upwards of 30 minutes in some cases?

Screenshot_2025-05-21_at_11.50.43_AM.png

MMarak Thank you, that helps. Do you have any idea why the wall time is so high for our...

Chaika•5/21/25, 10:15 PM

Is it just a specific durable object instance or is it across more then one?
Your code just updates the local storage, nothing else? hmm

CChaika Is it just a specific durable object instance or is it across more then one? You...

Marak•5/21/25, 10:21 PM

We have a "buddylist" where each "buddy" get's their own Durable Object. When a "buddy" goes online / offline, we were performing a fetch for each DO instance on the "buddylist" to update the local state for the users on their buddylist.

Marak•5/21/25, 10:27 PM

So each individual user is getting their own Durable Object instance keyed on their user name; and then each user maintains a local DO state of "buddylist" which contains an array of other users with status.

Marak•5/21/25, 10:38 PM

We found something very interesting actually, there was a comma where a semicolon should have been....

await updateBuddyList(this, request, update),

await updateBuddyList(this, request, update),

instead of

 await updateBuddyList(this, request, update);

 await updateBuddyList(this, request, update);

I wonder if this could have been a potential issue causing the request to hang.

MMarak We found something very interesting actually, there was a comma where a semicolo...

1984 Ford Laser•5/21/25, 10:39 PM

oh man that would be really unfortunate if that caused the requests to hang :NotLikeThis:

LLarry Besides alarms, another strategy is to use WebSockets. Request comes in over Web...

Super Morris•5/21/25, 10:42 PM

hey @Larry , thanks for your idea & @lambrospetrou thanks for your help!
my use-case is actually a little more complicated in real life:

I get http request from a client. sometimes I have to respond immediately, sometimes after task1, sometimes after task3 (depening on the users choice)
most api tasks are very fast, however recently ai request can take minutes
Sometimes the client gets disconnected in between a request and I still have to make sure the response is handled and all tasks are processed
its very time sensitive, so I can't use queues
its a http request from a client, i cant change this one to websocked, its out of my hand

we tried just with a worker, however max duration after disconnect is 30 seconds and when the client disconnects in between a longer request, we have no way to gracefully finish the current task (wait for the response) and continue it in the background.

This is my current idea.... a worker as a gateway, connected to the DO via websocket and keep the DO alive until the response is processed... whenever the client disconnects, the websocket connection to the DO is killed, however we can still process the tasks in the background.
I hope this works :/

if someone has a better idea how to structure this, I would be so unbelievably thankful!

11984 Ford Laser oh man that would be really unfortunate if that caused the requests to hang :Not...

Marak•5/21/25, 10:43 PM

I disabled the troubled code path in production and was able to isolate the issue. I am going to redeploy now and see if the missing semicolon was indeed the problem. I'm not 100% certain our architecture is valid for what we are trying to accomplish.

MMarak I disabled the troubled code path in production and was able to isolate the issu...

1984 Ford Laser•5/21/25, 10:45 PM

The mindset change for working with DOs can take a bit to work your head around. My main point is to always do the main work in a Worker, then defer to the DO to read/write some stuff

11984 Ford Laser The mindset change for working with DOs can take a bit to work your head around....

Marak•5/21/25, 10:47 PM

Agreed. It took me a few weeks at least to wrap my head around the paradigm of working with DOs. I think our core issue for our app is that without access to the native pub/sub we are having to re-implement fan-out logic.

MMarak Agreed. It took me a few weeks at least to wrap my head around the paradigm of w...

1984 Ford Laser•5/21/25, 10:54 PM

That has definitely been a pain point for me too, when I want to have some data be pushed out to tens of thousands of DO instances, it starts to get a bit messy/intensive. I wish there was a way to like 'link' instances together almost like a foreign key in a table, where updating one field/table/whatever in a DO will automatically push to other linked DOs as part of the runtime

11984 Ford Laser That has definitely been a pain point for me too, when I want to have some data ...

Marak•5/21/25, 10:56 PM

Did you find what a reasonable limit is for pushing looped updates to groups of DO's? We started to see issues even at 100 DO's ( think "buddies" in a "buddylist" ).

To my understanding the pub/sub would be able to achieve this by letting each DO subscribe to a channel for events. I think queues may work as well. Have you had any success?

MMarak Did you find what a reasonable limit is for pushing looped updates to groups of ...

1984 Ford Laser•5/21/25, 10:58 PM

WebSockets would be the best way to achieve this, where the buddies would connect to an overarching instance that lisens for updates, and then pushes updates only to subscribed users. Look up using the

topic

topic

parameter when accepting a hibernated Websocket, then using

getWebSockets(topic)

getWebSockets(topic)

to get required buddies

1984 Ford Laser•5/21/25, 10:59 PM

Not sure if that makes sense. I have written code to simply write data to 10s of thousands of instances in a loop, you can do at least 1000 instances a second if the work being done is sufficiently small. There are attempts to build sharding logic, like in the

durable-utils

durable-utils

library

11984 Ford Laser WebSockets would be the best way to achieve this, where the buddies would connec...

Marak•5/21/25, 11:04 PM

I hear you. Isn't the issue with this approach that it won't scale well over 1000 connected ws clients? A single point of failure for the ws connections? Our current approach gives each user their own DO with websocket server for their buddylist.

We are using your described approach for each individual chatroom / chat context ( single websocket server that broadcasts chat history ).

MMarak I hear you. Isn't the issue with this approach that it won't scale well over 100...

1984 Ford Laser•5/21/25, 11:05 PM

Oh yes these were two separate approaches, in theory the actual WS limit is 32000 per instance, if all your broadcasting is simple chat messages or something then you should be fine to scale to thousands of users. Beyond that, I'd look at sharding logic, which is something that I think is being actively considered by the dev team

1984 Ford Laser•5/21/25, 11:07 PM

lambrospetrou (who is a dev) has built this https://github.com/lambrospetrou/durable-utils that implements some form of sharding. I haven't looked at it in depth yet

11984 Ford Laser lambrospetrou (who is a dev) has built this https://github.com/lambrospetrou/dur...

Marak•5/21/25, 11:25 PM

Thank you, I read through https://raw.githubusercontent.com/lambrospetrou/durable-utils/refs/heads/main/src/do-sharding.ts

This looks like a "poor man's pub/sub", appears to have no message queues, delivery guarantees, or persistent state syncing; apologies for the direct language. I think for this approach to work effectively you'd have to take a slightly more sophisticated approach. Feels like it would need to be implemented at CF level and not user-space. Would be interested to hear @lambrospetrou insights on this.

I understand this approach, we were close to implementing the same thing. I suspect our actual issue today may have been the misplaced colon as a semi-colon causing the request to hang for minutes. Deployed the fix and watching logs.

Did you apply for the CF pub/sub beta?

MMarak Thank you, I read through https://raw.githubusercontent.com/lambrospetrou/durabl...

1984 Ford Laser•5/21/25, 11:27 PM

Pub/Sub is basically deprecated as far as Im aware, they're not letting anyone in. Not sure if lambro can speak to internal dev discussions, but there's been implications that things like sharding for instances will be coming in the future

11984 Ford Laser Pub/Sub is basically deprecated as far as Im aware, they're not letting anyone i...

Marak•5/21/25, 11:27 PM

I heard the same about pub/sub, sad. Instance sharding would be interesting if they get the fan-out logic working well.

Marak•5/21/25, 11:28 PM

I am putting a 100 user limit on the buddylist ( for now ) to ensure we don't exceed 100 DO updates in the update buddy status loop. If you are telling me you've achieved higher number than 100 DOs updating, I believe we should be okay.

Marak•5/21/25, 11:29 PM

I'm going to roll back the alarm queue / drain updates as well. We tried over-engineering the issue to alleivate stress on the DO, I think that was misplaced.

MMarak I am putting a 100 user limit on the buddylist ( for now ) to ensure we don't ex...

1984 Ford Laser•5/21/25, 11:29 PM

I was able to use 1 DO instance to push to 10K instances, and was able to complete the operation in under 10 seconds or something

11984 Ford Laser I was able to use 1 DO instance to push to 10K instances, and was able to comple...

Marak•5/21/25, 11:29 PM

That's very good to know, thank you.

1984 Ford Laser•5/21/25, 11:30 PM

If you used multiple/shard instances to deliver to parts of the overall list, you could complete it way faster too. Thats the cool thing with DO, if

buddy-1

buddy-1

is undergoing too much load, just spin up

buddy-1-shard1

buddy-1-shard1

to help handle load, you would just need to design that logic yourself

11984 Ford Laser If you used multiple/shard instances to deliver to parts of the overall list, yo...

Marak•5/21/25, 11:34 PM

Any thoughts on state contention with many updates at once? We are storing the user's buddylist in a single object on the DO. Updates take the entire object array, update it, and then re-saves the entire array. I'm guessing this approach will eventually potentially lose data with concurrent updates? The alternative would be storing each buddy as a single property on the DO ( instead of having them all inside an array of objects )

1984 Ford Laser•5/21/25, 11:35 PM

Yeah each buddy should be a single property maybe, are you using KV? If so, use prefixes in key names to be able to access specific properties, or list with the prefix to pull list of properties

11984 Ford Laser Yeah each buddy should be a single property maybe, are you using KV? If so, use ...

Marak•5/21/25, 11:37 PM

Not using KV, considered it many times. The perceived issue with the KV was responsiveness. We wish to see a buddy come online almost immediately, upwards of 60 seconds for eventual consistency seemed not great. Cost is also an issue, we suspected that DO writes would be cheaper than KV writes at scale.

1984 Ford Laser•5/21/25, 11:37 PM

Sorry no I mean KV within the DO versus SQLite

1984 Ford Laser•5/21/25, 11:38 PM

Even if you have a SQLite DO, you can use KV and the reads/writes are charged as rows read/written

Marak•5/21/25, 11:38 PM

Ahh, I somewhat glazed over that part. Isn't the KV inside DO being replaced with SQLite? I wasn't clear on that. Do you have a suggestion?

1984 Ford Laser•5/21/25, 11:39 PM

Nah KV is still available, its just abstracted away with SQLite

1984 Ford Laser•5/21/25, 11:40 PM

KV is available on both KV and SQL backend DOs, but SQLite is not available on KV backend DOs. I would recommend starting all DOs with SQLite at this point anyway

Marak•5/21/25, 11:40 PM

I'm going to migrate away from the single array of objects per buddy list and use the single key per buddy with prefix; we were considering this.

This may not work actually, we still need a quick way to fetch the whole buddylist.

Thanks for the help and advice today @1984 Ford Laser. I'm going to figure this out. I think we solved our 10 million wall time issue...

1984 Ford Laser•5/21/25, 11:43 PM

buddylist-{username}

buddylist-{username}

or something, then you could have

buddylist-{username}-config

buddylist-{username}-config

if you needed per user config, and then do a

list()

list()

buddylist-

buddylist-

to pull all users

MMarak I'm going to migrate away from the single array of objects per buddy list and us...

1984 Ford Laser•5/21/25, 11:43 PM

Yeah a

list({prefix: 'buddylist-'})

list({prefix: 'buddylist-'})

will be relatively performant depending on size of the list