I am researching migrating to a D1 Database. I need one source of truth for the relational data. I w

I am researching migrating to a D1 Database. I need one source of truth for the relational data. I will be updating the database with a backend application. The frontend will be performing reads. I need everyone in the world to get the most up to date data. I read that D1 performs replication geographically so this doesn't seem like a good fit. I would rather there be a delay than a client get the wrong or outdated data. Is it possible to somehow force one persistent storage to be used world wide? Are durable objects a solution for this dilemma? Can I enforce everyone in the United States to somehow use one specific D1 Database?

LLearnNMake I am researching migrating to a D1 Database. I need one source of truth for the ...

boop•1/25/25, 8:24 AM

There aren't replicas on D1 yet

LLearnNMake I am researching migrating to a D1 Database. I need one source of truth for the ...

lambrospetrou•1/25/25, 10:39 AM

Can I enforce everyone in the United States to somehow use one specific D1 Database?

This applies both to D1 and DO. If you want to use different databases per region, you can create a few of them, and then use your worker request location (in the request headers) to decide where to route your queries (choosing the database/DO that you want). Needs a bit of code, but doable.

I read that D1 performs replication geographically so this doesn't seem like a good fit. I would rather there be a delay than a client get the wrong or outdated data.

Once D1 launches read replication you will still be able to get the latest data if needed, we will not break the default consistency. This would mean though that all your requests would need to reach the primary database (transparent to your code, we would do this for you probably), so you would basically use D1 as its current form forgoing the replication features.

More on this once it's ready and out though, replication is not released yet, so now any query you make to D1 goes to the same database, hence you always get the latest data.

森

森@VerseLink (RW)•1/25/25, 11:41 AM

is it faster to store a user's durable id (calculated from the user id) in D1, then query D1 to get the DO object faster; or is it faster to just use the user's id to get the DO?

森

森@VerseLink (RW)•1/25/25, 11:42 AM

and it is a good idea the cache the DO's id (store on client via JWT) or just get the DO via the user id everytime instead?

森森@VerseLink (RW)is it faster to store a user's durable id (calculated from the user id) in D1, t...

lambrospetrou•1/25/25, 1:36 PM

Definitely directly to the DO. If you go to D1 first, you are already reaching a remote database, so you might as well go to the DO directly.
Read https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/ to get some more insight.

森森@VerseLink (RW)and it is a good idea the cache the DO's id (store on client via JWT) or just ge...

lambrospetrou•1/25/25, 1:38 PM

There shouldn't be a noticeable difference between these once you have regular traffic. Using

idFromName

idFromName

has one extra hop the first time you query it but then it's cached in cloudflare's datacenters, so assuming you have regular requests, there shoudn't be a performance benefit caching the DO ID yourself.
Always benchmark though, and it always depends on your traffic. Don't optimize prematurely.

森

森@VerseLink (RW)•1/25/25, 1:42 PM

Thank you very much!

Let's say we allow users to look up any user's public metadata (for example

/user/{userid}

/user/{userid}

, would it be better if we first look up D1 to see if the user id exists, or directly get the user's DO via

idFromName()

idFromName()

, and see if it was previously initialized, destroy it (via

deleteAll()

deleteAll()

) if not?

I would of course benchmark this, but would like some pointers.

森森@VerseLink (RW)Thank you very much! Let's say we allow users to look up any user's public meta...

lambrospetrou•1/25/25, 1:50 PM

and see if it was previously initialized, destroy it (via deleteAll()) if not?

Why would you need to delete all if nothing has been initialized?
In this case you need to make your constructor readonly, not writing anything to storage unless needed.
If you do all your table initialization in the constructor, then this won't work nicely.

Anyway, it depends on your usecase. There is a similar discussion a few weeks ago in this group about initialization done in constructor or not for publicly accessible links.

森

森@VerseLink (RW)•1/25/25, 2:19 PM

Got it, looking thru the documentation you provided, I have better understanding of what's happening behind the scene for DO.
Readonly as in it's acceptable to do

SELECT * FROM 'tablename'

SELECT * FROM 'tablename'

and we know the DO is not initialized if the table name doesn't exist? (or querying

sqlite_master

sqlite_master

to check if table exists)

森森@VerseLink (RW)Got it, looking thru the documentation you provided, I have better understanding...

James•1/25/25, 4:14 PM

DO object / stub initialization and table initialization are completely independent concepts. The constructor talk is just a convenience some developers take on.

You’ll always get errors querying tables don’t exist until you create them. How you manage migrations including table init is your call.

James•1/25/25, 4:16 PM

I personally find it computationally wasteful to constantly re-init or look for tables which may or may not be there on init. As in my app lifecycle, I may run millions or billions of requests on a DO between migrations I choose to create.

JJames DO object / stub initialization and table initialization are completely independ...

森

森@VerseLink (RW)•1/25/25, 4:23 PM

yeh i understand they are different concepts, i'm just concerned if i were to invoke any sql commands, it may not be readonly as I imagined

JJames I personally find it computationally wasteful to constantly re-init or look for ...

森

森@VerseLink (RW)•1/25/25, 4:23 PM

so your approach is to have a db lookup first then to invoke a DO?

James•1/25/25, 5:47 PM

The db is just a set of calls in the DO. You have to be in the DO to access it.

Aalgads When you say after the first hop, its cached in the local datacenter, is it cach...

lambrospetrou•1/26/25, 12:36 PM

There are too many variables in what you describe. How you instantiate DOs, how long that takes, is it serially or paralle, and many other things.
In general, any work you can parallelize from your worker the better. You should not pre-optimize DO stub creation until you actually measure it and see that it's the bottleneck in your code. Performance optimizations just based on intuition doesn't work. Measure and then optimize accordingly.
Start with simple code doing what you need and take it from there.

森森@VerseLink (RW)so your approach is to have a db lookup first then to invoke a DO?

lambrospetrou•1/26/25, 12:39 PM

My mental model for Durable Objects constructor is different than most.
As the step 3 in https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/ describes, I prefer doing the expensive DO initialization in a dedicated

init()

init()

call that is part of a resource creation flow. Including any initial SQL table creation and setup. After the init() completes, any other SQL migration is done only when necessary (at the point of access, not the constructor, and it's very fast/noop majority of the time when done properly: https://github.com/lambrospetrou/durable-utils?tab=readme-ov-file#sqlite-schema-migrations ).

I don't consider the constructor to be the place to do expensive initialization. The DO constructor as of today should be fast, and I only use it to initialize the "in-memory state of the DO". Loading anything from the storage into memory if necessary or just not do anything expensive.
It's different from how most folks think about the DO constructor, but in practice this works much better, especially if your DOs are not receiving enough traffic to stay warm and you have many cold starts, so you need to keep your constructor lightweight.

Llambrospetrou My mental model for Durable Objects constructor is different than most. As the s...

森

森@VerseLink (RW)•1/26/25, 2:26 PM

Thanks, yeh, I'm using Drizzle to manage migration of our database structure (just that, still write raw SQL most of the time), and the migration code it provided was less ideal so I had to incorporate your suggestion into it.

I guess premature optimization is evil, but I'm still curious about how I can approach DO init a bit better. I sort of understood how DO creates objects that were previously created (say via idFromName), but I'm still uncertain how DO creates when it's a new name id that it had never seen before. Does it ask every data center if that name id has existed before creating the object? or does it create it first anyways, and only sync at storage level?

If it was the latter, if two people decides to create the same name id DO at the same time on the opposite side of the earth, they would not see the same in memory status as the other? (or is there more memory syncing magic involved)

森森@VerseLink (RW)Thanks, yeh, I'm using Drizzle to manage migration of our database structure (ju...

lambrospetrou•1/27/25, 10:37 AM

Does it ask every data center if that name id has existed before creating the object? or does it create it first anyways, and only sync at storage level?

This is my personal suggestion, but you shouldn't design your applications based on internal implementation.
If DOs internally ask every datacenter, or some of them, or use a central database, should not impact your designs. The end result for you and your application is how long would this process take. So, do that, measure it, and only if it's slow for your use-case start thinking about optimizations. Internal implementations can and will change over time.

If it was the latter, if two people decides to create the same name id DO at the same time on the opposite side of the earth, they would not see the same in memory status as the other? (or is there more memory syncing magic involved)

Read https://developers.cloudflare.com/durable-objects/platform/known-issues/
The uniqueness guarantee is strictly enforced when you access storage.

lemmih•1/27/25, 10:39 AM

Can I emit log messages from a Durable Object?

geg•1/28/25, 8:11 AM

Hi, can it be that runtime updates break hibernated websockets? My DO can still send messages out but incoming messages stop arriving at random times. I noticed that it doesn't happen over the weekends and as soon as the weekend is over it starts happening almost every day, sometimes every couple of hours on a given day. My DO code couldn't be simpler, message in -> message out. I strongly believe it's runtime updates at this point. Reconnecting the "writing" websocket fixes it.

Ggeg Hi, can it be that runtime updates break hibernated websockets? My DO can still ...

lambrospetrou•1/28/25, 8:26 AM

Not sure about this specific issue, but you should be doing reconnection anyway, since servers can go down for many reasons, therefore your clients should attempt to reconnect (with usual jitter+backoff) if the connection fails.

Llambrospetrou Not sure about this specific issue, but you should be doing reconnection anyway,...

geg•1/28/25, 8:29 AM

What I meant with "messages stop arriving" is that there is a connection with no errors, yet messages don't arrive.

geg•1/28/25, 8:31 AM

When this issue happens to me, reconnecting doesn't work instantly also, the websocket connects fine but messages don't arrive still. I have to give it like 10 seconds before reconnecting, then magically it starts working again..

geg•1/28/25, 8:35 AM

When I deploy new code, the websocket disconnects and my "writer/sender" gets a close frame and reconnects fine.

nejc•1/28/25, 8:46 AM

hi all, one thing that I always prevent me from using DO is observability. Each object is isolated via ID, so can I get a whole picutre of what is happening. Let's say that I build a chat app which uses DO and each session ID is it's own DO ID. Now I would like to get real time overview of how many sessions I have and how many users are active. Is there an easy way to do this inside DO, or do I need D1 / Analytics engine in parallel, where DO pushes this data there?

iLearn•1/28/25, 9:25 AM

Pulling this Q out of thread (thread : Creating Replica for a DO), for kind attention and response :

Use-Case :
I have DO per user, accessed through unique_id (= email of user).
Now, a user requests to change email.
Hence, need to "replicate" complete DO(old_email) to DO(new_email)

My Q :
Q#1 : Is my request to "replicate" feasible ?
Q#2 : If yes, how to achieve this ? (a sample code will help)

Thanks.

IiLearn Pulling this Q out of thread (thread : Creating Replica for a DO), for kind atte...

lambrospetrou•1/28/25, 10:14 AM

There is no built in method now to duplicate the storage of a DO to a different one. You need to do the copy of the data in your code. Reading rows/KV items and writing them in the new one.

In my opinion though, if users are meant to be able to change their emails in your application, then those emails are not a good key to use for your DOs. The DO keys should be immutable identifiers for the data hosted in that DO. Otherwise it will become very expensive and slow to duplicate DOs all the time, and unless you block access to them while that's done you will also have consistency issues.

Ggeg What I meant with "messages stop arriving" is that there is a connection with no...

1984 Ford Laser•1/28/25, 12:17 PM

Are you broadcasting the message to all connected clients, or only the one that sent the message to the server?

1984 Ford Laser•1/28/25, 12:23 PM

If you're sending it directly, I'd be curious to see if the

ws

ws

can be found in the

this.state.getWebSockets()

this.state.getWebSockets()

set

11984 Ford Laser Are you broadcasting the message to all connected clients, or only the one that ...

geg•1/28/25, 12:25 PM

webSocketMessage(source: WebSocket, data: string) {
  for (let ws of this.ctx.getWebSockets()) {
    if (ws == source) {
      continue
    }
    
    ws.send(data)
  }
}

webSocketMessage(source: WebSocket, data: string) {
  for (let ws of this.ctx.getWebSockets()) {
    if (ws == source) {
      continue
    }
    
    ws.send(data)
  }
}

Just like this

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

Burrito•1/28/25, 12:35 PM

So if

transaction

transaction

can be used over

transactionSync

transactionSync

even in SQLite DO, does that mean the purpose of

transactionSync

transactionSync

is merely to provide a sync API?

Burrito•1/28/25, 12:38 PM

Cool.

Burrito•1/28/25, 12:40 PM

It'd be nice if the KV API also had sync equivalents in SQLite DO.

Llambrospetrou There is no built in method now to duplicate the storage of a DO to a different ...

iLearn•1/28/25, 4:12 PM

Thanks for the detailed explanation.

Ggeg ```typescript webSocketMessage(source: WebSocket, data: string) { for (let ws ...

1984 Ford Laser•1/28/25, 8:38 PM

Yeah okay that disproves my theory. I was wondering if the client had closed the connection unexpectedly after sending a message, meaning it wouldn't be in

getWebSockets()

getWebSockets()

, but your code proves it always would be for the server to respond. Interesting issue

iLearn•1/28/25, 10:14 PM

QQ : Can "Storage per Durable Object" for "key-value storage backend" store 200GB of data ? .. Limits section says "Unlimited", but still want this to be confirmed .

peachneo•1/28/25, 11:19 PM

What happens in terms of retries if you have an alarm that is triggered and that alarm handler also sets an alarm, folloowed by a failure:

eg:

async alarm() {
  await setAlarm(someTimestampFarIntoTheFuture)
  await somethingThatWillFail()
}

async alarm() {
  await setAlarm(someTimestampFarIntoTheFuture)
  await somethingThatWillFail()
}

do the retries get blown away by the new alarm time?

codefrau•1/28/25, 11:22 PM

It looks like the

durable_objects.namespaces.objects

durable_objects.namespaces.objects

API always returns

hasStoredData: false

hasStoredData: false

for SQL-backed DOs, even if they do have data. Is this intentional or a bug?

Ppeachneo What happens in terms of retries if you have an alarm that is triggered and that...

森

森@VerseLink (RW)•1/28/25, 11:28 PM

no i don't think so, you can image that there's a try catch call around

alarm()

alarm()

, so whatever happens within the

alarm()

alarm()

doesn't really affect how the original

alarm()

alarm()

is invoked

森森@VerseLink (RW)no i don't think so, you can image that there's a try catch call around `alarm()...

peachneo•1/28/25, 11:37 PM

Ok thanks, so effectively the alarm would run for N retries + the additional alarms set in the handler I guess?

Ppeachneo Ok thanks, so effectively the alarm would run for N retries + the additional ala...

lucidplot•1/28/25, 11:39 PM

no you can only have 1 alarm time set. If your alarm handler throws, it will retry as if you haven’t made any changes to stored state (ie forgetting any storage calls, including setAlarm)

Llucidplot no you can only have 1 alarm time set. If your alarm handler throws, it will ret...

peachneo•1/28/25, 11:39 PM

Oh ok.. that makes sense

Will•1/29/25, 2:32 AM

Does anyone have a resource on testing Durable object methods? I am using DO SqlStorage in my DO and hono to manage my worker.

I would be happy to just test the DO methods atm

IiLearn QQ : Can "Storage per Durable Object" for "key-value storage backend" store 200G...

1984 Ford Laser•1/29/25, 3:36 AM

This is the limit for total account storage (across all instances of all DOs) with Durable Objects, but you can contact CF to ask for a raise in that limit

11984 Ford Laser This is the limit for total account storage (across all instances of all DOs) wi...

iLearn•1/29/25, 6:29 AM

Thanks.
Can i have same luxury of Workers KV in terms of Unlimited Storage ? OR i need to regularly keep tap on storage of DO-KV and dropping mail to increase ?

WWill Does anyone have a resource on testing Durable object methods? I am using DO Sql...

lambrospetrou•1/29/25, 10:40 AM

You can use the vitest integration and test at the DO stub level directly: https://github.com/cloudflare/workers-sdk/blob/main/fixtures/vitest-pool-workers-examples/durable-objects/test/direct-access.test.ts

Llambrospetrou You can use the vitest integration and test at the DO stub level directly: https...

Will•1/30/25, 2:03 AM

Thank you!

Fernando Barroso•1/30/25, 2:26 AM

Hello all! Do you know when the SQLite storage will become GA? Or when the storage limit will be increased from 1GB? Will be a big decision point for us to use it in prod given that we do expect to have >1GB of data in our DOs

I am researching migrating to a D1 Database. I need one source of truth for the relational data. I w

Similar Threads

Similar Threads

Similar Threads