Had a hunch that might be the case... is it something the team's looking at or is it something that'

Had a hunch that might be the case... is it something the team's looking at or is it something that's impossible (or somewhere in between)?

Sscruffy Had a hunch that might be the case... is it something the team's looking at or i...

lambrospetrou•1/31/25, 1:31 PM

It's not impossible, since you can list all your KV items too if needed. https://developers.cloudflare.com/durable-objects/api/storage-api/#list

森

森@VerseLink (RW)•1/31/25, 5:34 PM

@lambrospetrou When calling transaction (asynchronous version), the DO deadlocks itself via blockConcurrencyWhile if the callback include a remote RPC stub that calls itself.

For example a function called “WriteData”, and an worker wants to coordinate two DOs and ensure both write the value at the same time, so both DO exposes a transaction method to allow this to happen. Result is that the DO deadlocks itself when the WriteData RPC stub is invoked inside the transaction.

I understand why this is the result, but I feel like if the caller initiated a transaction, they should also be able to do I/O too (and intuitively it should work this way too). Would you consider this a bug, or do you suggest a better way to approach this?

Best workaround would for the first DO to enter a transaction itself and call second DO to write and also enter a transaction itself, but this could require DO to expose too many functions that is only used for these transaction calls, and certainly more hard to catch bug (if you accidentally call the wrong function for example)

森森@VerseLink (RW)@lambrospetrou When calling transaction (asynchronous version), the DO deadlocks...

lambrospetrou•1/31/25, 5:42 PM

General guideline when you post, don't tag individuals please.

What you describe is the definition deadlock, so it makes sense that it deadblocks. blockConcurrencyWhileblockConcurrencyWhile is like a lock, nothing can get into your DO while that runs.

If you want to implement distributed transactions you need something like 2-phase commit, without blocking the DO itself (don't abuse use of blockConcurrencyWhileblockConcurrencyWhile). Store something in your state/storage that you are doing a transaction and reject anything if that's true.

I am working on a different transaction approach, for a personal project, but it's not ready for public yet, maybe in the future that could help you as well. But for now, distributed transactions need the standard 2-phase approach (or other techniques used by traditional systems).

In general, I don't like blockConcurrencyWhileblockConcurrencyWhile personally and try to avoid it.

Llambrospetrou General guideline when you post, don't tag individuals please. What you describ...

森

森@VerseLink (RW)•1/31/25, 5:51 PM

Oh sorry I’ll refrain from pinging.

With 2 phase commit approach, how do you rollback if one transaction fails on one DO, but success on another?

森森@VerseLink (RW)Oh sorry I’ll refrain from pinging. With 2 phase commit approach, how do you ro...

lambrospetrou•1/31/25, 5:54 PM

For 2PC to work there is one coordinator that controls that: https://en.wikipedia.org/wiki/Two-phase_commit_protocol#Commit_(or_completion)_phase

Eitherway, unless absolutely unavoidable, I suggest not doing distributed transactions unless you really get into these protocols. It will just slow down everything A LOT, and worst case lock you down.

If you need cross-DO coordination often, then maybe you need a different architectural design for your application.

Llambrospetrou For 2PC to work there is one coordinator that controls that: https://en.wikipedi...

森

森@VerseLink (RW)•1/31/25, 5:59 PM

Yeah…. I imagined this is what would need to be done, and I agree with not doing a lot of distributed transactions (but sometimes it could be unavoidable), I guess I was hoping for some alternative approach that I’m didn’t see.

Btw, transaction on KV api still invokes block concurrency through storage.transaction if I read the source code correctly right? Seems painful when the only async part is the KV storage api.

Isaac McFadyen•1/31/25, 10:28 PM

Asking here because it's probably the most related, do RPCs share subrequests with the caller?

Isaac McFadyen•1/31/25, 10:29 PM

As in, I'm wondering about both the 6 concurrent request limit and the 1000 request total limit.

geg•1/31/25, 11:37 PM

Can it be that hibernated websockets just stop working if there's no activity for a very long time?

geg•1/31/25, 11:38 PM

I keep running in to issues with people where the DO doesn't do anything even though websockets are connected...

Ggeg Can it be that hibernated websockets just stop working if there's no activity fo...

Milan•2/1/25, 12:10 AM

Do you have evidence of this happening? If so, can you give us code that reliably reproduces this behavior so we can verify it for ourselves?

MMilan Do you have evidence of this happening? If so, can you give us code that reliabl...

geg•2/1/25, 12:28 AM

Will let you know!

Ggeg I keep running in to issues with people where the DO doesn't do anything even th...

1984 Ford Laser•2/1/25, 5:23 AM

Are you sending directly to a single client, or multiple at once from server to client/s?

Gagan Suie•2/1/25, 5:45 AM

We have a userSession DO that handles websockets and chatRoom DO that handles DoSqlite. Is using fetch the only way to have them communicate? Wouldn't that cause micro letency every fetch?

1984 Ford Laser•2/1/25, 5:57 AM

Nah RPC will be your best bet for worker to worker/DO to DO invocations. The docs can be a bit lacking in describing this step, especially because the implementation can depend on how many calls you're making, which would make reusing your stubs more necessary (WorkerEntrypoint class vs RpcTarget class)

1984 Ford Laser•2/1/25, 5:58 AM

Downside usually is that if you've already written a bunch of code that executes through fetch then you may need to add/refactor a bunch of functionality

11984 Ford Laser Are you sending directly to a single client, or multiple at once from server to ...

geg•2/1/25, 8:30 AM

Observed with single and multiple clients receiving mesages via websocket
I supspect it might be that the client uses Bun runtime websockets, it's likely very buggy if I look at the issues on Bun github
Gonna test today with other websocket clients, shouldve done that way earlier

Llucidplot no you can only have 1 alarm time set. If your alarm handler throws, it will ret...

peachneo•2/1/25, 9:03 AM

Just some thoughts on this - can we effectively think of the alarm handler as being wrapped in a transaction?

Wondering if this is documented somewhere... the docs are a bit unclear on this from what I've read

peachneo•2/1/25, 9:11 AM

https://blog.cloudflare.com/durable-objects-alarms/

Under the hood, Alarms are implemented by making reads and writes to the storage layer. This means Alarm getget and setset operations follow the same rules as any other storage operation

The DO alarm's blog post references the storage blog post:

https://blog.cloudflare.com/durable-objects-easy-fast-correct-choose-three/#part-2-output-gates

From what I understand of this blog post, the following would be how things would play out? Which doesn't seem to align with <any failure within the alarm handler causes all alarm calls in that call to be ignored and for the original to be retried>

alarm() {
  await setAlarm() // This would be committed (and therefore overwrite the initial alarm? what happens to retries?)
  await someOtherStorageOperation() // this too given no failure up until the first fetch
  await fetch(...external) // Output - gates mean we now wait for previous operations to succeed

  await setAnotherAlarm() // This would not be committed because of subsequent failure (write coalescing)
  await triggerSomeFailure()
}

alarm() {
  await setAlarm() // This would be committed (and therefore overwrite the initial alarm? what happens to retries?)
  await someOtherStorageOperation() // this too given no failure up until the first fetch
  await fetch(...external) // Output - gates mean we now wait for previous operations to succeed

  await setAnotherAlarm() // This would not be committed because of subsequent failure (write coalescing)
  await triggerSomeFailure()
}

Aalgads OK, let me make sure I understand this. If I set a DO to invoke an alarm, the a...

peachneo•2/1/25, 9:18 AM

It seems this is an ongoing point of confusion - would be great for one of the CF devs to provide some insight here

Ppeachneo https://blog.cloudflare.com/durable-objects-alarms/ > Under the hood, Alarms ar...

lambrospetrou•2/1/25, 10:14 AM

First of all await setAnotherAlarm()await setAnotherAlarm() does not set a new alarm, it overrides the alarm you set in the first line.
There is only 1 alarm set at a time.

Regarding the behavior of setting an alarm and throwing right after, I will check with the team and get back to you.

Ggeg Observed with single and multiple clients receiving mesages via websocket I sups...

Milan•2/1/25, 5:40 PM

Definitely drop an update here if it turns out to be a client issue! If ws hibernation is broken we'd want to fix it for y'all asap. FWIW I haven't seen other reports of this and we've tested this in the past, but not continuously testing this so it's certainly possible something has broken since we last tried it. I can't think of any changes to that code in quite some time though...

ps2000•2/1/25, 8:33 PM

Hello! I'm new to CF. Working on a DO with WSs. I'm wondering if it is possible to reassign a socket to a different DO after an ws.accept()? The reason: I'd like to "shard" the ws connections but I won't know the shard key until the first ws message. (first server message is a "connect" message with an auth key which the shard key is derived from).

And wondering how many ws connections can. be maintained per DO given the 128MB memory limit? I can estimate my own per ws state (eg: 3KB).

11984 Ford Laser You will need to do a `state.storage.deleteAll()` to remove an old instance. An ...

Eric•2/2/25, 6:08 AM

state.storage.deleteAll()state.storage.deleteAll() happens per-instance. Each user I have has an access to his/her own instance, but for some users, I no longer have access to user ID (forgot to track them...). Aren't they still going to incur a storage charge?

EEric `state.storage.deleteAll()` happens per-instance. Each user I have has an access...

森

森@VerseLink (RW)•2/2/25, 7:32 AM

yes, they will incur storage charge

森

森@VerseLink (RW)•2/2/25, 7:32 AM

if you forgot to track them, you can technically find a list of DOs here: https://developers.cloudflare.com/api/resources/durable_objects/subresources/namespaces/methods/list/

Cloudflare API | Durable Objects › Namespaces › List Namespaces

Interact with Cloudflare's products and services via the Cloudflare API

森

森@VerseLink (RW)•2/2/25, 7:33 AM

but it's using the cloudflare API and has stricter rate limitations

MMilan Definitely drop an update here if it turns out to be a client issue! If ws hiber...

geg•2/2/25, 10:03 AM

yep nvm, it's because I didn't send protocol level pings..

Ggeg yep nvm, it's because I didn't send protocol level pings.. 🤡

quan•2/3/25, 2:47 AM

hi i may have same issue, I was trying the default worker-chat-demo from github, is it just a blank msg send from client that you did to get over that?

Ggeg yep nvm, it's because I didn't send protocol level pings.. 🤡

1984 Ford Laser•2/3/25, 3:17 AM

not necessarily a question for you directly, but as someone who doesn't know websockets well, is there a suggested interval where we should be sending/acking pings to optimally keep clients connected?

DiamondDragon•2/3/25, 3:24 AM

I am playing around with drizzle's durable object integration. Got it working. How are people inspecting their DO storage? Is there any chance an integration with drizzle studio might be coming soon?

DiamondDragon•2/3/25, 3:27 AM

I'd also be curious to learn more about how people are building apps where you have a central database (say postgres) and then you utilize DOs. Are people persiting the DO namespace/ID in their main app DB and then splitting off other types of data to keep in the DO?

Like if you were to do any analytical querying, this data is scattered around all these DO's. Im just curious if there's been any writing anyone has done on these new ways of thinking when finding areas to implement a DO in an app, maybe progressively

DDiamondDragon I'd also be curious to learn more about how people are building apps where you h...

1984 Ford Laser•2/3/25, 5:57 AM

currently I'm just aggregating the different data from different DOs within a process then using it as necessary. If aggregated data doesn't change often, and gets accessed regularly, then it makes sense to save the aggregated data and disperse as necessary. That aggregated data could in theory be its own DO which handles itself as necessary. Service Bindings and RPC can make things easier to handle depending on how you need to handle the data

geg•2/3/25, 6:42 AM

@quan @1984 Ford Laser It was definitely because I didn't send pings when not sending any messages, It's a mistake on my part

I was so used to browser environment where that part is abstracted that I just forgot about it completely. I haven't tested how long the ping interval can be yet, would be nice if documented

elunesarrow•2/3/25, 9:31 AM

Hi, I need some clarifications regarding DO billing:

[1] "Each Durable Object stays alive as long as requests are being processed, and remains alive for several seconds after being idle before hibernating"
[2] "Once your Object finishes responding to all requests, it will stop incurring duration charges"
So assume if for my use-case, I only use RPC to query/migrate sqlite, does this mean I will get duration charges only for the RPC session, which is probably a few milliseconds, or also for the whole "alive duration" which includes "several seconds after being idle before hibernating"?

[1] https://developers.cloudflare.com/durable-objects/what-are-durable-objects/#durable-objects-highlights
[2] https://developers.cloudflare.com/durable-objects/platform/pricing

Eelunesarrow Hi, I need some clarifications regarding DO billing: - [1] "Each Durable Object ...

1984 Ford Laser•2/3/25, 9:49 AM

Only the few milliseconds is billed, the few seconds of aliveness is mostly a runtime consideration and doesn't really matter for most development architectures, unless you need to persist the DO state specifically between calls

jed•2/3/25, 12:22 PM

hey all, i've got a setup in which a worker serves an app (html / js) and an api (websocket). to prevent app deployment from resetting connections, i've got the app on a service binding. my question is this: what's the best way for the api to report app deployment events in realtime? if outgoing DO websockets could hibernate, i'd have the app server host a websocket that just reported its version on connection, but given that's not possible, what's the easiest way to achieve this? i'd rather not use the DO list API or open two websockets from the browser... would long polling from DO to app server be viable? i'd like the events to be as realtime as possible to allow instant reloads. thanks!

Ggeg @quan @1984 Ford Laser It was definitely because I didn't send pings when not se...

quan•2/3/25, 2:07 PM

oh, how long do you do the interval now? i am trying a timeout of 15 seconds to send a ws.send("") from react client to the websocket server, then server response back

i had more success with SSE before, but not sure why on ws it keeps disconnecting me from the server after around 30seconds

Qquan oh, how long do you do the interval now? i am trying a timeout of 15 seconds to ...

geg•2/3/25, 2:15 PM

If your client is in the browser this shouldn't be necessary. I was talking about websocket pings (not messages).

Rahul gangotri•2/3/25, 3:22 PM

I am running a browser rendering that goto a url and fetch text content from the page inside a durable object

i also reuse sessions and pick any free random session to decrease the load time

still many url's are taking upto 15 sec. i am waiting until networkidle0

can someone please help me optimize this?

Rahul gangotri•2/3/25, 3:23 PM

when i used to run playwright on my own machine and scrape using that, it takes 2-3 sec for page to load

Rahul gangotri•2/3/25, 3:23 PM

same page here is taking 10-15 sec.

DDiamondDragon I'd also be curious to learn more about how people are building apps where you h...

James•2/3/25, 4:06 PM

I read aggregate data a lot more than write transactional, so I usually aggregate to central DB on writes to source tables (in DO) in real time.

For example, rather than query all DOs for some cross DO action like, 'last updated TS', I'll sync up a central log of all 'last updated TS' mapped to the DO.

Llambrospetrou First of all `await setAnotherAlarm()` does not set a new alarm, it overrides th...

lambrospetrou•2/3/25, 7:00 PM

Circling back to this, if in your alarm handler you set a new alarm, and your handler throws AFTER, there won't be a retry since a new alarm has been successfully set.

Best practice for alarms, always set the new alarm AFTER your processing is done, so that exceptions are retried properly, if you need the retries.

RRahul gangotri I am running a browser rendering that goto a url and fetch text content from the...

Rahul gangotri•2/3/25, 10:56 PM

Anyone help on this?

Nnil wrote a util to help with maximum 6 concurrent network requests... has anyone do...

lambrospetrou•2/3/25, 11:53 PM

I have a similar, but simpler approach in my DO helper package: https://github.com/lambrospetrou/durable-utils/blob/main/src/do-sharding.ts#L154-L187

That's a bit specific to the specific class logic, but it does the same thing as yours, limit processing among N things with a given concurrency. Check it out in case it simplifies anything on your side.
I didn't know about the p-limitp-limit package you used though, so maybe that's enough already.

Had a hunch that might be the case... is it something the team's looking at or is it something that'

Similar Threads