message.txt3.48KB
TIf I delete a durable object namespace from console, will that also wipe all instances that have been created and the data they hold?
PMaybe try wrapping the prisma operations in the storage transaction api instead ? https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction. My guess is this is a problem with drizzle orm and not durable objects ? There's 2 issues that seem related https://github.com/drizzle-team/drizzle-orm/issues/4322 https://github.com/drizzle-team/drizzle-orm/issues/4202
Ti did a workaround, but its very ugly!
i thought drizzle is popular ORM to use with CloudFlare durableObject.
i thought drizzle is popular ORM to use with CloudFlare durableObject.
message.txt2.78KB
PSqlite storage in DO is something very new, it barely ended its beta phasis few weeks ago. Drizzle adopted it fairly quickly, only few weeks ago the typing for DO sqlite wasn't even supported with the orm and now it is, so I feel they are working as fast as they can to adapt accordingly. Give them a little bit of time, DO's a special creature that needs the libs to work a little differently. I wouldn't be surprised your problem may be addressed in a forseable future with user adoption and bug reports.
JTheir DO driver has some issues, and unfortunately they don't seem to be paying much attention to it
JI have a PR open since December to fix a bug on it and its still sitting there https://github.com/drizzle-team/drizzle-orm/pull/3746
GitHub
Returns the correct value from Durable Objects SQLite transactions, which were always returning an empty object regardless of the value returned by the callback.
JI use drizzle purely to generate migrations for DO. Then I reuse the drizzle schema at runtime for type checking post-query responses. All actual DO request logic is pure SQL. Has worked pretty well for nearly a year now.
FIf there are any people here interested in sync engines and building a library around durable objects feel free to send me a message. So far ive taken inspiration from the amazing Zero team: https://zero.rocicorp.dev
SIs there a way in DO to wait for a response longer than 30seconds after client disconnected?
Modern API Request take longer than 30 seconds and I need to make sure that the response is processed.
(without queue)
Modern API Request take longer than 30 seconds and I need to make sure that the response is processed.
(without queue)
LYou can use Alarms to keep the DO alive, or simply doing
setTimeoutso you say, as long as I keep it alive with an alarm it is able to handle the response of an http request?
LBesides alarms, another strategy is to use WebSockets. Request comes in over WebSockets. Optionally you confirm receipt of the request over the WebSocket. However, you push the request to a Worker (with a longer stay-alive time and cpu time billing instead of DOs clock time billing) that accomplishes the request. When it's done, it calls back to the DO that wakes up the hiberated WebSocket and sends the response.
WThis API endpoint seems to be busted
https://api.cloudflare.com/client/v4/accounts/:account_id/workers/durable_objects/namespaces/:namespace/objectsI reported it to the team and they will take a look. Thanks for reporting.
MSomething seems very wrong with our Durable Objects today; we are seeing 10,000,000+ ms wall times on requests that take < 1ms CPU. Concerned this may lead to a catastrophic billing situation.
M^^^ Can anyone recommend the best way to query the current Durable Objects Compute Duration usage for an account? Trying to get a handle on this situation.
CUnder Compute (Workers) -> Durable Objects, you can see your account usage(request, duration, storage usage, etc)
MThank you, that helps. Do you have any idea why the wall time is so high for our Durable Objects? The affected instances shouldn't have any other connected clients or persistent connections. We are performing an RPC from another service to update state inside the DO. CPU time is minimal, but the wall time is holding for upwards of 30 minutes in some cases?
CIs it just a specific durable object instance or is it across more then one?
Your code just updates the local storage, nothing else? hmm
Your code just updates the local storage, nothing else? hmm
MWe have a "buddylist" where each "buddy" get's their own Durable Object. When a "buddy" goes online / offline, we were performing a fetch for each DO instance on the "buddylist" to update the local state for the users on their buddylist.
MSo each individual user is getting their own Durable Object instance keyed on their user name; and then each user maintains a local DO state of "buddylist" which contains an array of other users with status.
MWe found something very interesting actually, there was a comma where a semicolon should have been....
I wonder if this could have been a potential issue causing the request to hang.
await updateBuddyList(this, request, update), await updateBuddyList(this, request, update);I wonder if this could have been a potential issue causing the request to hang.
1oh man that would be really unfortunate if that caused the requests to hang :NotLikeThis:
Shey @Larry , thanks for your idea & @lambrospetrou thanks for your help!
my use-case is actually a little more complicated in real life:
This is my current idea.... a worker as a gateway, connected to the DO via websocket and keep the DO alive until the response is processed... whenever the client disconnects, the websocket connection to the DO is killed, however we can still process the tasks in the background.
I hope this works :/
if someone has a better idea how to structure this, I would be so unbelievably thankful!
my use-case is actually a little more complicated in real life:
- I get http request from a client. sometimes I have to respond immediately, sometimes after task1, sometimes after task3 (depening on the users choice)
- most api tasks are very fast, however recently ai request can take minutes
- Sometimes the client gets disconnected in between a request and I still have to make sure the response is handled and all tasks are processed
- its very time sensitive, so I can't use queues
- its a http request from a client, i cant change this one to websocked, its out of my hand
This is my current idea.... a worker as a gateway, connected to the DO via websocket and keep the DO alive until the response is processed... whenever the client disconnects, the websocket connection to the DO is killed, however we can still process the tasks in the background.
I hope this works :/
if someone has a better idea how to structure this, I would be so unbelievably thankful!
MI disabled the troubled code path in production and was able to isolate the issue. I am going to redeploy now and see if the missing semicolon was indeed the problem. I'm not 100% certain our architecture is valid for what we are trying to accomplish.
1The mindset change for working with DOs can take a bit to work your head around. My main point is to always do the main work in a Worker, then defer to the DO to read/write some stuff
MAgreed. It took me a few weeks at least to wrap my head around the paradigm of working with DOs. I think our core issue for our app is that without access to the native pub/sub we are having to re-implement fan-out logic.
1That has definitely been a pain point for me too, when I want to have some data be pushed out to tens of thousands of DO instances, it starts to get a bit messy/intensive. I wish there was a way to like 'link' instances together almost like a foreign key in a table, where updating one field/table/whatever in a DO will automatically push to other linked DOs as part of the runtime
MDid you find what a reasonable limit is for pushing looped updates to groups of DO's? We started to see issues even at 100 DO's ( think "buddies" in a "buddylist" ).
To my understanding the pub/sub would be able to achieve this by letting each DO subscribe to a channel for events. I think queues may work as well. Have you had any success?
To my understanding the pub/sub would be able to achieve this by letting each DO subscribe to a channel for events. I think queues may work as well. Have you had any success?
1WebSockets would be the best way to achieve this, where the buddies would connect to an overarching instance that lisens for updates, and then pushes updates only to subscribed users. Look up using the
topicgetWebSockets(topic)Not sure if that makes sense. I have written code to simply write data to 10s of thousands of instances in a loop, you can do at least 1000 instances a second if the work being done is sufficiently small. There are attempts to build sharding logic, like in the
durable-utilsI hear you. Isn't the issue with this approach that it won't scale well over 1000 connected ws clients? A single point of failure for the ws connections? Our current approach gives each user their own DO with websocket server for their buddylist.
We are using your described approach for each individual chatroom / chat context ( single websocket server that broadcasts chat history ).
We are using your described approach for each individual chatroom / chat context ( single websocket server that broadcasts chat history ).
1Oh yes these were two separate approaches, in theory the actual WS limit is 32000 per instance, if all your broadcasting is simple chat messages or something then you should be fine to scale to thousands of users. Beyond that, I'd look at sharding logic, which is something that I think is being actively considered by the dev team
1lambrospetrou (who is a dev) has built this https://github.com/lambrospetrou/durable-utils that implements some form of sharding. I haven't looked at it in depth yet
MThank you, I read through https://raw.githubusercontent.com/lambrospetrou/durable-utils/refs/heads/main/src/do-sharding.ts
This looks like a "poor man's pub/sub", appears to have no message queues, delivery guarantees, or persistent state syncing; apologies for the direct language. I think for this approach to work effectively you'd have to take a slightly more sophisticated approach. Feels like it would need to be implemented at CF level and not user-space. Would be interested to hear @lambrospetrou insights on this.
I understand this approach, we were close to implementing the same thing. I suspect our actual issue today may have been the misplaced colon as a semi-colon causing the request to hang for minutes. Deployed the fix and watching logs.
Did you apply for the CF pub/sub beta?
This looks like a "poor man's pub/sub", appears to have no message queues, delivery guarantees, or persistent state syncing; apologies for the direct language. I think for this approach to work effectively you'd have to take a slightly more sophisticated approach. Feels like it would need to be implemented at CF level and not user-space. Would be interested to hear @lambrospetrou insights on this.
I understand this approach, we were close to implementing the same thing. I suspect our actual issue today may have been the misplaced colon as a semi-colon causing the request to hang for minutes. Deployed the fix and watching logs.
Did you apply for the CF pub/sub beta?
1Pub/Sub is basically deprecated as far as Im aware, they're not letting anyone in. Not sure if lambro can speak to internal dev discussions, but there's been implications that things like sharding for instances will be coming in the future
MI heard the same about pub/sub, sad. Instance sharding would be interesting if they get the fan-out logic working well.
MI am putting a 100 user limit on the buddylist ( for now ) to ensure we don't exceed 100 DO updates in the update buddy status loop. If you are telling me you've achieved higher number than 100 DOs updating, I believe we should be okay.
MI'm going to roll back the alarm queue / drain updates as well. We tried over-engineering the issue to alleivate stress on the DO, I think that was misplaced.
1I was able to use 1 DO instance to push to 10K instances, and was able to complete the operation in under 10 seconds or something
MThat's very good to know, thank you.
1If you used multiple/shard instances to deliver to parts of the overall list, you could complete it way faster too. Thats the cool thing with DO, if
buddy-1buddy-1-shard1Any thoughts on state contention with many updates at once? We are storing the user's buddylist in a single object on the DO. Updates take the entire object array, update it, and then re-saves the entire array. I'm guessing this approach will eventually potentially lose data with concurrent updates? The alternative would be storing each buddy as a single property on the DO ( instead of having them all inside an array of objects )
1Yeah each buddy should be a single property maybe, are you using KV? If so, use prefixes in key names to be able to access specific properties, or list with the prefix to pull list of properties
MNot using KV, considered it many times. The perceived issue with the KV was responsiveness. We wish to see a buddy come online almost immediately, upwards of 60 seconds for eventual consistency seemed not great. Cost is also an issue, we suspected that DO writes would be cheaper than KV writes at scale.
1Sorry no I mean KV within the DO versus SQLite
1Even if you have a SQLite DO, you can use KV and the reads/writes are charged as rows read/written
MAhh, I somewhat glazed over that part. Isn't the KV inside DO being replaced with SQLite? I wasn't clear on that. Do you have a suggestion?
1Nah KV is still available, its just abstracted away with SQLite