atm you'd have to do what D1 does: write your own sqlite dump script (we based ours on sqlite's CLI

atm you'd have to do what D1 does: write your own sqlite dump script (we based ours on sqlite's CLI - https://github.com/sqlite/sqlite/blob/105c20648e1b05839fd0638686b95f2e3998abcb/src/shell.c.in#L8463-L8469)

GitHub

sqlite/src/shell.c.in at 105c20648e1b05839fd0638686b95f2e3998abcb ...

Official Git mirror of the SQLite source tree. Contribute to sqlite/sqlite development by creating an account on GitHub.

Vvilvei they have an interesting wording for value, on DO storage put: "The value can be...

lambrospetrou•4/11/25, 10:55 AM

If you click the link referenced you get to https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm that lists the types supported and not supported.

MMax (@rozenmd)atm you'd have to do what D1 does: write your own sqlite dump script (we based o...

gruntlord6•4/11/25, 11:09 AM

So wrangler just runs that in the background in d1?

MMilan Honestly, might be worth opening an issue in the workers-sdk or workerd github r...

boop•4/11/25, 11:17 AM

https://www.answeroverflow.com/m/1339728309646524458 likely a workflows issue rather than a DO issue

Ggruntlord6 So wrangler just runs that in the background in d1?

Max (@rozenmd)OP•4/11/25, 12:09 PM

D1 runs it in the background, wrangler calls out to the service that runs it to grab the data

gruntlord6•4/11/25, 12:11 PM

Good to know, I appreciate the transparency

marko.333•4/11/25, 3:16 PM

Is there any chance that Sqlite with DO SQL Storage (+D1 for consistency) could be enhanced with the (existing) sqlite-vec extension? With the extension already available, it seems like a modest-effort super-capability to filter/rank by embeddings (optionally combined with fulltext FTS5 (more agent/ai buzz you can announce;-)
If anyone is remotely interested - please vote for this request

https://community.cloudflare.com/t/please-add-support-for-sqlite-vec-for-durable-objects-sql-storage-and-d1/786935

veeque•4/11/25, 3:30 PM

what happens if i try to write to the sql db whilst it is full? does it throw an exception?

gruntlord6•4/11/25, 3:36 PM

I would imagine it would be the same as if you hit a request limit or network error, it would throw an error and restart the object

gruntlord6•4/11/25, 3:37 PM

I've never come close to using 10gb of space though

veeque•4/11/25, 3:38 PM

for my usecase i might eventually hit it, but if i can have a way to handle the write failing it wouldn't be a problem

Vveeque what happens if i try to write to the sql db whilst it is full? does it throw an...

lambrospetrou•4/11/25, 3:50 PM

I suggest you be proactive here and check the databaseSizedatabaseSize periodically or after your queries and act/alert/error-log when you are close to the limit. You are definitely going to get an error at some point when it's full.
I wouldn't rely on that though for error handling within your DO code.

veeque•4/11/25, 3:59 PM

ok, locally i didn't get any error. my idea is to "shard" when it's full. i don't expect to hit that limit, but could happen for some DOs if the user makes heavy use of it.

i would like to attempt to write, catch the error, and just pop a new DO with /1/1 or /2/2 appended to it, and do the write there. i would have a "hasMorePages"-like variable that is set to true if this scenario ever happened. this way i could paginate and yes, would make queries slower since i would have to chain hop the next shards.

is this too dumb?

gruntlord6•4/11/25, 4:00 PM

Ideally you would shard proactively so your data flow works with that mindset out the gate

gruntlord6•4/11/25, 4:00 PM

Like having each user have a database vs a central database with all that data for example

veeque•4/11/25, 4:05 PM

it's storing a bunch of uuids and some data (a big list of "tickets". the queries i need to do are "does it exist" or "which is john's ticket"). each of my customers can make "events", which each would get a DO. most would have less than 500 tickets, but those 2-3 customers will definetly exceed the about 6mln tickets 10gb can handle. the idea would be that "if the shard flag is toggled, then you should keep looking for another". i would generate DOs with this id: {user-id}/0{user-id}/0 so if you know it's sharded you'd attempt {user-id}/1{user-id}/1 and so on

veeque•4/11/25, 4:06 PM

i can't store in r2 since i need to query inside the ticket too

gruntlord6•4/11/25, 4:49 PM

could you just have an object that indexes the objects and then store the actual data sharded?

gruntlord6•4/11/25, 4:50 PM

then you could quickly locate where things are and where they should be stored rather then checking one object and then checking another

gruntlord6•4/11/25, 4:50 PM

Just spitballing, my data was designed with sharding in mind but I've not needed to shard older data sets yet

DDanny What was your change?

thelunz•4/11/25, 10:37 PM

Apologies for going silent on this. The issue - and homing in on a workaround that actually works - has taken all this time.

The root of the problem is that console logs written from a hibernatable-websocket DO during a wakeup that was triggered by a websocket message don't get through to logpush (we opened https://github.com/cloudflare/workerd/issues/2089 on this last July; it's still open). Our solution to that had been to send all log output from such a DO to a separate Logging worker, originally using a fetch but recently with an RPC invocation.

What we hadn't noticed until a couple of weeks ago was that something about that external-logging setup was keeping the main DO awake full time - every second of every day. Turning it off is what immediately collapsed our duration costs... but also meant we were missing all those log messages.

We now have a different workaround, which writes logs generated during websocket-triggered wakeups into persistent store, and dumps them out to the console on the next non-websocket wakeup. There were some nuances in juggling how often to write to storage, and how to ensure that a non-websocket wakeup would happen in a sufficiently timely manner to avoid the stored logs growing indefinitely. As of now, we seem to be in a good place: we're getting all the log messages, and our duration metric is under control.

GitHub

🐛 Bug Report — Logging broken in hibernatable WebSocket event ...

We have one DurableObject that uses the hibernation API. It implements webSocketMessage() in which we use console.log() to log stuff. On miniflare (wrangler dev) that works perfectly fine, the logs...

Tthelunz Apologies for going silent on this. The issue - and homing in on a workaround t...

Danny•4/12/25, 1:43 AM

Interesting find. How do you distinguish between websocket wakeups and normal (rpc/fetch?) ones?
I am missing some lifecycle callbacks here.

LordSilver•4/12/25, 7:48 AM

why you need a DO to connect to a container? why cannot connect from a normal worker?

Tthelunz Apologies for going silent on this. The issue - and homing in on a workaround t...

Milan•4/12/25, 3:33 PM

I think some work was started internally to fix this but got deprioritized, let me see if we can just get this re-prioritized and fixed.

veeque•4/13/25, 1:58 PM

with sveltekit i cannot (yet) directly code a durable object in the same project as the frontend/backend worker. to use durable objects in this case tho what would you recommend? should i make a worker that acts as kind of a "internal api" or is there a way to make just the DO, and link it even if they are in two separate projects?

Vveeque with sveltekit i cannot (yet) directly code a durable object in the same project...

James•4/13/25, 2:30 PM

Yeah I use a monorepo to create both workers. One just for the DO, then I’ll use the fetch handler on the DO worker locally to handle the bindings.

zehawk•4/13/25, 3:03 PM

How does one test when a DO is in hibernation? I have just migrated the WebSocket Standard API to WebSocket Hibernation API - now how would I test whether this is really working? And would love to see stats around it in the dashboard.

Zzehawk How does one test when a DO is in hibernation? I have just migrated the WebSocke...

1984 Ford Laser•4/13/25, 9:12 PM

Well if you let it sit there connected not doing anything, your GB-sec usage should be nothing

11984 Ford Laser Well if you let it sit there connected not doing anything, your GB-sec usage sho...

zehawk•4/14/25, 5:29 AM

Yup that's one way and I am monitoring

, but was wondering about something a bit more exact, like being able to log on some condition.

Tero•4/14/25, 6:45 AM

Hi, I have a usecase where I run websocket connections on a durable object. Some of the messages I receive require making external API calls. These external calls are fairly slow (1-2 sec).

I don't want to block the durable object for that duration, so I thought that the durable object could invoke the worker (which also deploys the DO) without awaiting for the response, and then the worker would call back to the DO once the work is completed. But I can't figure how to make a binding that would allow the DO to call the same worker that deployed it.

I could make the call with the full DNS name that points to the worker, but then I will need to go through our CF access protection and that feels like a pointless complication.

So in short, is there a way to make a binding to "self" so that the durable object could call (rpc or fetch) the worker that deployed the DO.

TTero Hi, I have a usecase where I run websocket connections on a durable object. Some...

1984 Ford Laser•4/14/25, 7:37 AM

You just make a service binding which references the same worker

11984 Ford Laser You just make a service binding which references the same worker

Tero•4/14/25, 8:38 AM

Yeah that is what I tried to do, but I did not get it to work. When wrangler starts up locally it shows that binding as disconnected, and when I try to call it, the call also fails. I did not find any explicit mentions from the documentation if this should be possible or not, so if someone has a working example I am curious to hear / see about that one.

TTero Yeah that is what I tried to do, but I did not get it to work. When wrangler sta...

1984 Ford Laser•4/14/25, 8:40 AM

Sorry I dont have much experience with testing locally, what command are you running?

11984 Ford Laser Sorry I dont have much experience with testing locally, what command are you run...

Tero•4/14/25, 8:52 AM

In local env I run it with wrangler dev --env dev wrangler dev --env dev and all other bindings (durable objects, other workers, queues) work as expected.

And in my wrangler.toml I tried to have

[[env.dev.services]]
binding = "MY-BINDING"
service = "my-worker-name-dev"

[[env.dev.services]]
binding = "MY-BINDING"
service = "my-worker-name-dev"

1984 Ford Laser•4/14/25, 10:15 AM

Does adding --remote help at all?

Tero•4/14/25, 11:31 AM

I also use CF Queues in my setup and they don't support --remote, so not too keen to start exploring that option... https://developers.cloudflare.com/queues/configuration/local-development/#known-issues

thelunz•4/14/25, 6:07 PM

Our DO now has an in-memory wakeupTypewakeupType property, that we start as the empty string. The possible reasons for a wakeup are (1) a fetch, (2) a websocket message, or (3) an alarm. Each of these comes through a single entry point (whereas allowing incoming RPC calls muddies that, so we've put those back to being fetches). When any of those points is hit while the wakeupTypewakeupType is still empty we set the type accordingly, and that determines how we handle log output for the rest of that wakeup.

Tthelunz Our DO now has an in-memory `wakeupType` property, that we start as the empty st...

thelunz•4/14/25, 7:04 PM

sorry - this was meant to be a reply to @Danny above. Somehow I fumbled the reply connection.

Matthew•4/14/25, 9:44 PM

Hey, has anybody done any measurement/comparison between D1 and Hyperdrive performance wise? I'd like to migrate from D1 to Hyperdrive and would love to know more beforehand.
Thanks Xx

MMatthew Hey, has anybody done any measurement/comparison between D1 and Hyperdrive perfo...

1984 Ford Laser•4/14/25, 9:55 PM

I dont think theyre directly comparable products, especially as Hyperdrive performance could be affected by the origin DB. Might be better to ask in #d1-database or #hyperdrive

MMatthew Hey, has anybody done any measurement/comparison between D1 and Hyperdrive perfo...

1984 Ford Laser•4/14/25, 9:56 PM

?crossposting

SuperHelpflare•4/14/25, 9:56 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

TTyveris I have this exact same problem. My current solution has been to have the worker ...

Tero•4/15/25, 5:12 AM

I actually experimented with queues but realised they had another problem with a consumer delay that was too slow for my use case, tried to ask about that one here: queues

veeque•4/15/25, 8:59 AM

I am using Durable Objects to make a service for an app since my model allows me to instantiate a DO per user, this way I don't run into the 10GB limit a single D1 database would have. The only issue is that some power users will definitely exceed this limit and I'll have to shard the data across multiple instances.

When writing to the underlying SQL database I don't know how much space the inserted data will take up. I would like a way to:

start something like a transaction;
attempt to insert data;
if succeeded, return;
if failed (due to storage cap), rollback, and move to next shard.

I have the sharding figured out. I'm missing the rollback/transaction and knowing what error is thrown by DO. I know from reading somewhere that for D1 it's a D1_ERROR but even that is not documented.

I really would like not to have a "attempt to write, if it's 0.9GB move to another shard just in case" scenario.

Rocket Hosting•4/15/25, 9:20 AM

Need help All domain records gone but cloudflare is showing them
Any idea?