Is there any advice from the Cloudflare team around how to import a SQL dump file into a Durable Obj

Is there any advice from the Cloudflare team around how to import a SQL dump file into a Durable Object (with SQLite backend, obviously) when the dump is larger than the memory limit for a Durable Object (128 MB)? I can't think of how this could be achieved (within a single transaction, that is). To be clear - I don't currently have this problem... but I'm trying to figure out if it's going to be one in the future. Thanks!

JJacob Marshall Is there any advice from the Cloudflare team around how to import a SQL dump fil...

lambrospetrou•2/21/25, 7:34 AM

You can use streams to process the request without having the whole file in memory. However, doing hundreds of MBs or GBs will probably need you to split it in smaller chunks otherwise you will hit cpu/time limits.
Example of streaming a file in DO: https://github.com/lambrospetrou/tiddlyflare/blob/main/src/durable-objects.ts#L281

Llambrospetrou You can use streams to process the request without having the whole file in memo...

Jacob MarshallOP•2/21/25, 8:20 AM

Cheers. Yeah, I figured I'd have to do it async (with streams). Hopefully someday it'll be possible without that though

JJacob Marshall Cheers. Yeah, I figured I'd have to do it async (with streams). Hopefully someda...

lambrospetrou•2/21/25, 8:21 AM

I doubt that. You will always have less memory than what people need for bulk imports. Streaming is the way to go.

Llambrospetrou I doubt that. You will always have less memory than what people need for bulk im...

Jacob MarshallOP•2/21/25, 8:22 AM

A man can dream can't he haha

Matt•2/21/25, 6:32 PM

So as I understand it, if you use storage.put()storage.put() multiple times without awaiting, it will group all the writes into a single round trip. If you

put

put

to the same key multiple times without awaiting, will it write to that key multiple times or will it only write the last one?

TTyveris So with durable objects, if you plan to make api calls, it seems like its best t...

hunt•2/22/25, 3:19 AM

An approach you could consider is using a hybrid of rest and websockets. Instead of trying to create one layer of abstraction, you let rest do the thing it's good at which is handling transactional requests, and let websockets handle what it's good at which is streaming data.

Your clients would subscribe to an "event stream" DO with a user id, and when you're handling transactional requests and need to push an event to any client, you call the DO with the event and user id which then pushes that off to the client

atgctg•2/22/25, 11:40 AM

Is there a better way to make a value available in a DO's constructor? This is the only way I found:

app.get('/avatars', async c => {
  const user = c.get('user')
  c.env.uid = user.uid // is this a bad practice?
  const userObjectId = c.env.USER_OBJECT.idFromName(user.uid)
  const userObject = c.env.USER_OBJECT.get(userObjectId)
  const avatars = await userObject.getAvatars()
  return c.json(avatars)
})

app.get('/avatars', async c => {
  const user = c.get('user')
  c.env.uid = user.uid // is this a bad practice?
  const userObjectId = c.env.USER_OBJECT.idFromName(user.uid)
  const userObject = c.env.USER_OBJECT.get(userObjectId)
  const avatars = await userObject.getAvatars()
  return c.json(avatars)
})

Aatgctg Is there a better way to make a value available in a DO's constructor? This is t...

lambrospetrou•2/22/25, 1:06 PM

I doubt this works. Did you test it? You are modifying a local variable here, not the actual env injected into your worker/DO if it happens to be created in a different machine or even a different isolate/process.
If you want data in your DO that are not in the envenv, you need to pass them in your RPC call to the DO.

Llambrospetrou I doubt this works. Did you test it? You are modifying a local variable here, no...

atgctg•2/22/25, 1:09 PM

The variable shows up on env, but only tested locally.
Just would have been nice to be able to initialize it in the constructorconstructor, I might make an

init

init

method so I don't have to pass it on every call

Aatgctg The variable shows up on env, but only tested locally. Just would have been nice...

lambrospetrou•2/22/25, 1:11 PM

See https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/

Cloudflare Docs

Control and data plane architectural pattern for Durable Objects · ...

Durable Objects are built on-top of Cloudflare Workers spanning several locations across our global infrastructure network.
Each Durable Object instance has its own durable storage persisted across requests, in-memory state, single-threaded execution, and can be placed in a specific region.

Llambrospetrou See https://developers.cloudflare.com/reference-architecture/diagrams/storage/du...

atgctg•2/22/25, 1:23 PM

That's a great resource, thanks.
So based on that, I think I should call init(uid)init(uid) the first time the user object is created, save the user's id to storage, and this way it will be available in storage whenever I'm calling that DO again, right? Just need to set it in the constructorconstructor like this:

ctx.blockConcurrencyWhile(async () => {
  this.uid = await ctx.storage.get('uid')
})

ctx.blockConcurrencyWhile(async () => {
  this.uid = await ctx.storage.get('uid')
})

Aatgctg That's a great resource, thanks. So based on that, I think I should call `init(u...

James•2/22/25, 4:11 PM

I’m conservative on rpc calls and to avoid unnecessary round trips, I just pass a metadata obj to all my rpc calls as an optional arg which defaults some helpful initialization stuff to the DO

1984 Ford Laser•2/23/25, 7:32 AM

Anyone know if partysub has a limit on the number of topics that can be specified?

TTyveris Specific questions: 1. Is it ok to pull the websocket property off the internal ...

geg•2/23/25, 5:21 PM

This is how it's done, you're on the right track

TTyveris It seems like it is possible to create a WebSocket connection from a worker to a...

Franco Romano Losada•2/23/25, 6:39 PM

Have you tried PartyServer (https://github.com/threepointone/partyserver/tree/main). You avoid a lot of boilerplate and get some good things too. I was doing it like do but from Hono but this library is nice, and provides a nice client too

chronark•2/24/25, 9:25 AM

anyone else getting a ton of errors right now from durable objects?

- Internal error in Durable Object storage write caused object to be reset.Internal error in Durable Object storage write caused object to be reset.
- Network connection lost.Network connection lost.

1984 Ford Laser•2/24/25, 10:43 AM

Why would this code cause 2 DO read operations when a websocket client connects then immediately disconnects?

11984 Ford Laser Why would this code cause 2 DO read operations when a websocket client connects ...

Isaac McFadyen•2/24/25, 11:04 AM

You're checking path but not method I expect

Isaac McFadyen•2/24/25, 11:04 AM

I assume that you're trying to connect a WebSocket for the initial OPTIONS preflight and that's causing the extra request

Isaac McFadyen•2/24/25, 11:04 AM

Assuming WS sends preflight

IIsaac McFadyen You're checking path but not method I expect

1984 Ford Laser•2/24/25, 11:04 AM

The path check is in the Worker not the DO

11984 Ford Laser The path check is in the Worker not the DO

Isaac McFadyen•2/24/25, 11:05 AM

Yeah but if it's an OPTIONS preflight the DO will still be spawned because of the get() and fetch()

Isaac McFadyen•2/24/25, 11:05 AM

From the Worker

1984 Ford Laser•2/24/25, 11:06 AM

Im connecting using Postman, haven't noticed any OPTIONS requests in the real time logs

Isaac McFadyen•2/24/25, 11:07 AM

Hmmm OK maybe not that then

1984 Ford Laser•2/24/25, 11:07 AM

I also respond to options preflight earlier in the worker prior to pulling a stub

1984 Ford Laser•2/24/25, 11:10 AM

Also shouldn't a DO read operation only happen when accessing storage, not pulling the stub itself?

11984 Ford Laser Also shouldn't a DO read operation only happen when accessing storage, not pulli...

lambrospetrou•2/24/25, 12:58 PM

Yes, just getting the stub and going into the DO doesn't cause any storage operation.
From the snippet you shared, I am not sure if acceptWebSocket()acceptWebSocket() does any storage operation, like reading something and updating it with the new socket, since it needs to somehow persist that.

Llambrospetrou Yes, just getting the stub and going into the DO doesn't cause any storage opera...

Milan•2/24/25, 1:44 PM

acceptWebSocket()acceptWebSocket() doesn't do any storage operations (see hibernation-manager.c++).

GitHub

workerd/src/workerd/io/hibernation-manager.c++ at 2f7d61d69f5066836...

The JavaScript / Wasm runtime that powers Cloudflare Workers - cloudflare/workerd

cloudkite•2/24/25, 7:35 PM

Is there any documentation/guidance around what happens when you update your DO script. Found https://community.cloudflare.com/t/durable-objects-state-during-lifecycle-events/319932/5 does that same restriction apply when using this.ctx.storage.sql?

cloudkite•2/24/25, 7:38 PM

Would be great if ongoing requests are given chance to complete, rather than throwing error when storage API is accessed, as this will cause breakages in our usecase.

Unsmart•2/24/25, 8:12 PM

That message still applies. Durable objects are supposed to only have one instance so when code is updated it gets a new instance and the storage is moved to the new instance and no longer accessible by the old instance otherwise it can break things in unexpected ways.

cloudkite•2/24/25, 8:32 PM

That’s a shame, in the case of SQLite where you control the schema with migrations, i feel like allowing storage ops for inflight requests has much less potential to break things unexpectedly than throwing an error

cloudkite•2/24/25, 8:39 PM

Also the majority of code updates might not have any changes to the schema

Ccloudkite That’s a shame, in the case of SQLite where you control the schema with migratio...

lambrospetrou•2/24/25, 9:20 PM

If you run your queries within transactions (implicit or explicit) then nothing should be broken.

Ashley•2/24/25, 11:59 PM

I haven't tried this outside of local, but based on the documentation I was expecting this.ctx.id.namethis.ctx.id.name to return me the string used when calling const stub = env.MY_DO.idFromName('foo')const stub = env.MY_DO.idFromName('foo') - it works fine if I try it on the stub (e.g. stub.namestub.name), but it doesn't work inside the DO itself if I try this.ctx.id.namethis.ctx.id.name - it's undefined. The ID itself is there, but considering the type is of DurableObjectIdDurableObjectId I expected to be able to get the name too - is this not the case?

AAshley I haven't tried this outside of local, but based on the documentation I was expe...

Unsmart•2/25/25, 12:04 AM

No you cannot get the name from inside the DO. You would need to pass the name through and save to storage

Ashley•2/25/25, 12:07 AM

Guessing there is a technical reason why it's not populated? As on the face of it, it is quite annoying it's not populated and (unless I missed it) the docs aren't explicit that it won't be set, and I sort of hoped it would be based on the type

Unsmart•2/25/25, 12:10 AM

The name gets hashed into an id so the DO doesnt know what the name was it only knows the id

Ashley•2/25/25, 12:14 AM

Yeah I get that, I am curious what the point of having

name

name

available on there at all is really considering it can only be called right after you retrieved a DO with said string

(but it's also very late so maybe my tired brain isn't seeing something)

Walshy•2/25/25, 12:34 AM

https://github.com/cloudflare/workerd/issues/2240

Llambrospetrou If you run your queries within transactions (implicit or explicit) then nothing ...

cloudkite•2/25/25, 12:54 AM

I need to run async things like vectorize, openai interleaved with sql queries though? transactions only work if its all sync right?

Iggy•2/25/25, 3:19 AM

Is the 50GB SQLite storage per account a hard limit or soft? If it’s soft, what range would enterprise expect to be able to get to?

HowlOftheSun•2/25/25, 4:25 AM

If I use a decorator on a function in a do would the decorator be able to write to storage in the do? Would I have to pass the ctx?

Ccloudkite I need to run async things like vectorize, openai interleaved with sql queries t...

lambrospetrou•2/25/25, 8:04 AM

I don't follow. Why would you be doing SQL data migrations while you invoke remote APIs? You can do async transactions too, BTW, but it's going to be horribly slow if you mix many migrations with remote APIs.
Anyway, depends what you are doing, but there are async transactions: https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction

Is there any advice from the Cloudflare team around how to import a SQL dump file into a Durable Obj

Similar Threads

Similar Threads

Similar Threads