Thanks - well actually things work as expected locally, just not in prod

EElliot Hesp Thanks - well actually things work as expected locally, just not in prod

L

lambrospetrou•5/9/25, 3:24 PM

Describe the issue here, then. Without knowing what the discrepancy is I can't help.

E

Elliot HespOP•5/9/25, 3:32 PM

Ok so it does trigger in production

I think some code is erroring but I've no idea where logs are if an alarm throws an error

EElliot Hesp Ok so it does trigger in production 😮‍💨 I think some code is erroring but I've...

L

lambrospetrou•5/9/25, 3:39 PM

You can enable Observability logs for your worker project and you will be able to see your logs in the cloudflare dashboard. https://developers.cloudflare.com/workers/observability/logs/workers-logs/#enable-workers-logs

E

Elliot HespOP•5/9/25, 3:43 PM

Hmm, I do have that enabled, but I've just seen what's happening:

I have an endpoint setup which I am calling which accesses the DO (to logs the alarm)
If I don't call this endpoint, the alarm doesn't trigger
If I then call the endpoint, the alarm triggers - so accessing the DO seems to kick it off

E

Elliot HespOP•5/9/25, 3:43 PM

And I can see the RPC call which kicks it off includes the logs of the alarm call

EElliot Hesp Hmm, I do have that enabled, but I've just seen what's happening: - I have an e...

M

Milan•5/9/25, 4:14 PM

Maybe this? durable-objects

E

Elliot HespOP•5/9/25, 4:19 PM

@lambrospetrou @Milan

Ok so finally figured it out, I think:

My alarm calls a function

await doSomething()

await doSomething()

await doSomething()

await doSomething() which is outside of the class (import function) which does an external async fetch call.

If that function thows an error (e.g.

if (!res.ok) throw new Error()

if (!res.ok) throw new Error()

if (!res.ok) throw new Error()

if (!res.ok) throw new Error()) then the alarm just dies and no logs appear in console (hence me thinking it wasn't being triggered)

If I catch the error inline within the alarm

await doSomething().catch(e => console.error(e))

await doSomething().catch(e => console.error(e))

await doSomething().catch(e => console.error(e))

await doSomething().catch(e => console.error(e)) and return early, the alarm logs show up

So my issue was that I didn't think the alarm was being triggered because the alarm swallowed the errors/logs

EElliot Hesp @lambrospetrou @Milan Ok so finally figured it out, I think: My alarm calls a...

L

lambrospetrou•5/9/25, 4:24 PM

Yeah, as a guideline anything you do inside the

alarm()

alarm()

alarm()

alarm() handler should be wrapped in a try-catch.

E

Elliot HespOP•5/9/25, 8:44 PM

Ok I've boiled the issue down to the following:

    this.ctx.blockConcurrencyWhile(async () => {
      await this.migrate();
    });

    this.ctx.blockConcurrencyWhile(async () => {
      await this.migrate();
    });

    this.ctx.blockConcurrencyWhile(async () => {
      await this.migrate();
    });

    this.ctx.blockConcurrencyWhile(async () => {
      await this.migrate();
    });

This is taken from; https://orm.drizzle.team/docs/connect-cloudflare-do

Seems like when the alarm tries to init the DO, it just explodes for some reason trying to the migration

Drizzle ORM - Cloudflare Durable Objects

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

E

Elliot HespOP•5/9/25, 8:46 PM

The above was just a red herring it seems

J

jcheese•5/9/25, 11:03 PM

Is there a way to “re-instantiate” a specific do?

Jjcheese Is there a way to “re-instantiate” a specific do?

1

1984 Ford Laser•5/9/25, 11:10 PM

What do you mean by that exactly? you can delete its storage which essentially resets it

11984 Ford Laser What do you mean by that exactly? you can delete its storage which essentially r...

J

jcheese•5/9/25, 11:14 PM

theres some DOs that just give "internal error; reference = moh644b953d9vckab1seqm9a" and I cant access it at all

J

jcheese•5/9/25, 11:14 PM

like i want to delete them and re-make it from scratch

1

1984 Ford Laser•5/9/25, 11:14 PM

Is that just by running the constructor? What is the instance trying to do when it errors like that

1

1984 Ford Laser•5/9/25, 11:15 PM

You can only delete them from within

V

Victor•5/10/25, 12:26 AM

Does anyone know the exact error message when a SQLite DO exceeds the 10gb of storage?

V

Victor•5/10/25, 1:02 AM

I made a random generator and filled a local DO, but it appears wrangler local doesn't simulate the 10GB cap

S

sathoro•5/10/25, 1:23 PM

my guess is RangeError

P

planet•5/10/25, 4:13 PM

could look at databaseSize

A

aza547•5/10/25, 6:51 PM

Hello, I'm looking at using Durable Objects as a Websocket server. I'm a bit confused by the Hibernate API, and this SO post.

Since DurableObjects are basically in-memory classes, their long-term state cannot be guaranteed - especially in cases which involve the Hibernation API. When the DurableObject hibernates, the existing websocket connections are kept alive "outside" of the class - and when one wakes up, the Workers runtime will reinstantiate the DurableObject with a new state.

My confusion is that I thought the whole point of DO was to be a "stateful worker". But this comment seems to claim the opposite (might just be bogus). That said, the docs also claim that:

// ... During periods of inactivity, the Durable Object can be evicted
// from memory, but the WebSocket connection will remain open. If at some later point the
// WebSocket receives a message, the runtime will recreate the Durable Object
// (run the
constructor
constructor
constructor
constructor) and deliver the message to the appropriate handler.

Am I misunderstanding the point of DOs? Or is this just an edge case with the hibernate API and I should use

serializeAttachment

serializeAttachment

serializeAttachment

serializeAttachment /

deserializeAttachment

deserializeAttachment

deserializeAttachment

deserializeAttachment to store persistent state?

Aaza547 Hello, I'm looking at using Durable Objects as a Websocket server. I'm a bit con...

U

Unsmart•5/10/25, 7:48 PM

Durable objects are not exclusive in memory storage and can absolutely store stuff permanently (that is indeed the entire point of them wouldnt be durable otherwise).

The attachments on a websocket connection will exist as long as the client is connected regardless of the DO being evicted and then reinstated down the line. However, if the client disconnects then the attachment will go away.

You can use memory only state that doesnt get durably stored if you wanted but you can also use durable storage.

A

aza547•5/10/25, 9:58 PM

Ah I see. Seems the bit I was missing is that that durable storage is separate from the memory state. RTFM I guess. Thanks for the help.

V

veeque•5/11/25, 2:24 PM

if a durable object is "hot" (so if i was to make a new request now the constructor would not run) and I deploy a new version in the meantime, would the constructor run again (kinda like "shutting down" the DO)?

Wwilb.me Hi! I'm trying to figure out if a singleton Durable Object makes sense as a cons...

1

1984 Ford Laser•5/12/25, 9:52 AM

If its 24/7 operation then yeah VPS will probably be best at the moment, Containers are coming to CF :soontm: which might be a better solution that's on-platform

Wwilb.me Thanks! I was hoping some magic around "idle but active websocket connection" wo...

1

1984 Ford Laser•5/12/25, 10:00 AM

My hope is that using DOs as a hibernatable client will be possible in the future, but nothing specific has been hinted at yet. I'm super keen for containers, I have so many interesting use cases

Wwilb.me Hi! I'm trying to figure out if a singleton Durable Object makes sense as a cons...

L

Larry•5/12/25, 1:19 PM

Have you considered using the "backfill window" capability of ATProto combined with an alarm that is shorter than that window to accomplish the same thing. I'm not familiar with Jetstream, but maybe it supports this.

Alternatively, you could use CBOR-X over WebSockets to subscribe directly to ATProto. It seems like the only thing Jetstream provides is converting CBOR to JSON, but that's what CBOR-X does as a library instead of a service.

E

echoes221•5/12/25, 3:13 PM

With durable object migrations, do I need to do migrations for each env? Or just a top level one?

V

veeque•5/12/25, 7:34 PM

is

this.ctx.storage.sql.exec("select * from artist where artistname = ?;", "Alice");

this.ctx.storage.sql.exec("select * from artist where artistname = ?;", "Alice");

this.ctx.storage.sql.exec("select * from artist where artistname = ?;", "Alice");

this.ctx.storage.sql.exec("select * from artist where artistname = ?;", "Alice"); safe against sql-injection or do i need to worry about that?

Llambrospetrou The `.exec()` method does the prepare statements for you. Soon they will be cach...

V

veeque•5/12/25, 7:36 PM

nvm found the answer

A

ashish•5/13/25, 12:34 AM

<just kidding, not a real question>

From the documentation:

The first time you get a Durable Object stub based on an ID derived from a name, the system has to take into account the possibility that a Worker on the opposite side of the world could have coincidentally accessed the same named Durable Object at the same time. To guarantee that only one instance of the Durable Object is created, the system must check that the Durable Object has not been created anywhere else. Due to the inherent limit of the speed of light, this round-the-world check can take up to a few hundred milliseconds.

The first time you get a Durable Object stub based on an ID derived from a name, the system has to take into account the possibility that a Worker on the opposite side of the world could have coincidentally accessed the same named Durable Object at the same time. To guarantee that only one instance of the Durable Object is created, the system must check that the Durable Object has not been created anywhere else. Due to the inherent limit of the speed of light, this round-the-world check can take up to a few hundred milliseconds.

The first time you get a Durable Object stub based on an ID derived from a name, the system has to take into account the possibility that a Worker on the opposite side of the world could have coincidentally accessed the same named Durable Object at the same time. To guarantee that only one instance of the Durable Object is created, the system must check that the Durable Object has not been created anywhere else. Due to the inherent limit of the speed of light, this round-the-world check can take up to a few hundred milliseconds.

The first time you get a Durable Object stub based on an ID derived from a name, the system has to take into account the possibility that a Worker on the opposite side of the world could have coincidentally accessed the same named Durable Object at the same time. To guarantee that only one instance of the Durable Object is created, the system must check that the Durable Object has not been created anywhere else. Due to the inherent limit of the speed of light, this round-the-world check can take up to a few hundred milliseconds.

It takes light around 50ms to cover the earth. Why does this check take several hundered milliseconds? So seems there are other limits than just speed of light ?

Cloudflare Docs

Durable Object Namespace

A Durable Object namespace is a set of Durable Objects that are backed by the same Durable Object class. There is only one Durable Object namespace per class. A Durable Object namespace can contain any number of Durable Objects.

Aashish <just kidding, not a real question> From the [documentation](https://developers...

S

SakuraFC•5/13/25, 2:59 AM

Speed of light in the fiber is 0.6 of the speed in vacuum, plus cables are not in a straight line, plus reads from disk and other processing === long delay. Also, round-trips, probably more than a few

P

planet•5/13/25, 4:42 AM

plus checking lava lamps

M

mon•5/13/25, 5:17 AM

Can't find this mentioned elsewhere - is there a timeline for when

wrangler dev --remote

wrangler dev --remote

wrangler dev --remote

wrangler dev --remote will support SQLite DOs?

SSakuraFC Speed of light in the fiber is 0.6 of the speed in vacuum, plus cables are not i...

A

ashish•5/13/25, 6:05 AM

Reads from disk + other processing makes total sense. I am just saying, lets not blame it all on the speed of light.

S

SakuraFC•5/13/25, 6:07 AM

Well, round trip for tcpip three way handshake will add a lot to that, plus any other round trips. The check is fast, I am sure

Aashish Reads from disk + other processing makes total sense. I am just saying, lets not...

P

planet•5/13/25, 7:52 AM

“has not been created anywhere else” “round-the-world”

defo not a distance reason

Aashish Reads from disk + other processing makes total sense. I am just saying, lets not...

B

boop•5/13/25, 8:09 AM

Speed of light is most definitely the primary factor. Here's a table of the network latency between the most popular DO locations. If the instantiating colo needs to confirm with all other colos, you're waiting for the slowest responder to finish.

https://where.durableobjects.live/ provides a map of all the DO colos. It also shows latency metrics between all edge POPs and DO colos.

Original message was deleted

B

boop•5/13/25, 8:20 AM

You got this clarified for us last time

B

boop•5/13/25, 8:20 AM

durable-objects

S

Silvan•5/13/25, 12:31 PM

im surprised theres still 0 DO POPs in south america

S

Silvan•5/13/25, 5:45 PM

i dont get your argument here

SSilvan i dont get your argument here

H

Hard@Work•5/13/25, 5:56 PM

For DOs to be deployed in a colo, there needs to be a certain number of datacenters within that colo, for redundancy. If a colo doesn't have enough DCs, it can't run DOs(or at least not without a modicum of disaster recovery)

HHard@Work For DOs to be deployed in a colo, there needs to be a certain number of datacent...

S

Silvan•5/13/25, 6:05 PM

interesting considering oceania barely has any while south africa has a lot? unless i misunderstood this https://where.durableobjects.live/colo

Where Durable Objects Live

Colos

Tracking where Durable Objects are created, wherever you are in the world.

SSilvan interesting considering oceania barely has any while south africa has a lot? unl...

H

Hard@Work•5/13/25, 6:07 PM

Not sure I understand what you mean? South Africa(the country) has 3 colos, while the general Oceania area has 11 colos, at least from a visual count

S

Silvan•5/13/25, 6:10 PM

sorry meant south america

isnt every circle a datacenter?

“For DOs to be deployed in a colo, there needs to be a certain number of datacenters within that colo”

all i see is that australia has like 7 with 3 of them supporting DOs
while brazil has like 15+ and no DO ones