Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably try and migrate thi

Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably try and migrate this code to Durable Objects to avoid the cross-region shenanigans - I've been told we tried and failed before, but with SQLite in DO now I think it'll be smooth sailing.

monOP•5/12/25, 6:43 AM

Well, not perfectly smooth sailing because [WARNING] SQLite in Durable Objects is only supported in local mode.[WARNING] SQLite in Durable Objects is only supported in local mode., but I can probably work around that for the time being

Mmon Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably...

lambrospetrou•5/12/25, 6:45 AM

BTW, the limit I mentioned applies to DOs as well, but at least there your code is near the database so you can do your main logic there to reduce the 5MB result further down before returning it out of the DO.

monOP•5/12/25, 6:46 AM

Yep, and that's exactly what my plan is - the rows are all read, lightly processed and saved to R2. By moving this work into the DO, we entirely eliminate the RPC overhead.

monOP•5/12/25, 6:47 AM

Plus it's information about a user session, so I get to simplify all the queries and just have one DO per session instead of filtering the rows.

monOP•5/12/25, 6:47 AM

Honestly it really is the perfect usecase for DOs and I'm sad it didn't work out the first time

ByteBeast•5/12/25, 11:10 AM

I have one table in which I have auth column when i query it into worker and fetch value in that column it gets converted in to **. what is this and how can I get actual value?

yinxingmaiming6409•5/13/25, 8:55 AM

I want to experience RealtimeKit. Can someone review it for me?

Yyinxingmaiming6409 I want to experience RealtimeKit. Can someone review it for me?

1984 Ford Laser•5/13/25, 11:43 AM

You need to be posting in #realtime

kchro3•5/13/25, 4:29 PM

Hey, I started seeing a bunch of D1_ERROR: Network connection lostD1_ERROR: Network connection lost (even after adding retries) where every write would fail.

the trace looked like this:

" at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1DatabaseSessionAlwaysPrimary.batch (cloudflare-internal:d1-api:80:23) at async Object.queue (index.js:6144:5)",

" at D1DatabaseSessionAlwaysPrimary._sendOrThrow (cloudflare-internal:d1-api:125:19) at async D1DatabaseSessionAlwaysPrimary.batch (cloudflare-internal:d1-api:80:23) at async Object.queue (index.js:6144:5)",

i was trying to use batch on a list of insert statements like this:

await env.D1.batch(
  batch.map((item) => stmt.bind(...))
);

await env.D1.batch(
  batch.map((item) => stmt.bind(...))
);

is there a soft limit on how large a batch can be? i was able to resolve it by breaking up the batches into smaller batches of 25, but it wasn't causing problems before and the full batch size is <100

Tuna•5/13/25, 6:59 PM

Hi,

I have workers, durableObject lets say one durableObject is for keep track of uploads from client to server (i have my own API for multipart upload), the second is for access to R2 and also used as life cycle, alarm is set for TTL and after that its deleting the file/s from R2.

Im wondering what will be best practice, i want the durableObject to sync R1, like when user upload files, it will keep track in durable object but also sync the global database R1 which will be per user, and i will need one database for all users.
i also have an api gateway worker with itty router to handle rest API.

should i update the D1 from the router handler / workers ? should i clean the R2 from the alarm?
or should i use queues?
if i use queues for async operations, they need to be bounded to the same worker or i can set the main worker to be producer and create a second worker to be the consumer ?

parisholley•5/13/25, 7:22 PM

anyone seeing a higher than usual amount of D1 errors (timeout, connection lost, etc)

kchro3•5/14/25, 1:02 AM

yes, i saw a huge spike of errors in the last week without changing the code at all

Pparisholley anyone seeing a higher than usual amount of D1 errors (timeout, connection lost,...

gruntlord6•5/14/25, 5:03 AM

yea had an automated job fail because it could not create a table this morning due to D1 timing out

parisholley•5/14/25, 2:17 PM

do reads effectively "lock" the d1 database? eg: if i'm doing an expensive read-only query that takes 1-2s, does this prevent other clients from doing reads?

Dean•5/15/25, 12:25 AM

i thought reads were multi-threaded but writes are single threaded

Dean•5/15/25, 12:26 AM

is dynamic bindings for D1 coming eventually? I've been working around it for a while but need to start to re-think it long term if it's never going to arrive

DDean is dynamic bindings for D1 coming eventually? I've been working around it for a ...

Hard@Work•5/15/25, 12:45 AM

SQLite DOs are kind of dynamic bindings? They have most of the features, minus read replicas, but with scripting next to the DB

HHard@Work SQLite DOs are kind of dynamic bindings? They have most of the features, minus r...

Dean•5/15/25, 12:50 AM

i get DO's exist, but i built the platform around D1 with dynamic bindings coming eventually (which is fine). I'm hesitant to change everything from D1 -> DO and then D1 ends up with dynamic binding support and I've wasted my time. Eventually i'd like these databases replicated so I'm sort of in limbo.

Deda•5/15/25, 1:05 AM

is it possible to connect to the d1 trought the node.js app locally, or its only available for the workers?

Chaika•5/15/25, 1:23 AM

?d1-remote-connections

SuperHelpflare•5/15/25, 1:23 AM

Accessing D1 from outside workers:

D1 is based on SQLite, which does not natively support remote connections and isn't MySQL/PostgreSQL wire compatible, nor does D1 itself offer that functionality. As such, your options are:

1 -> The Cloudflare API Query Endpoint (https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database), with the upside of needing less setup, all requests have to go back to Cloudflare's core, so request latency is far from optimal from some locations. You would also be subject to the API Rate limit of 1200 requests per 5 minutes.

2 -> Create a worker to proxy requests to your D1 Database. Either tightly bound to your application/schema or loose and just accepting any query. This would allow your worker to execute potentially closer to your D1 Database and avoid the api rate limit.

Ffrankichiro Hello! What is the correct API request to get the same info as `wrangler d1 info...

Chaika•5/15/25, 8:39 PM

Wrangler is using GraphQL
https://github.com/cloudflare/workers-sdk/blob/02d40ed3bbfc9cb4c2f95fb921efd7ec56f141a6/packages/wrangler/src/d1/info.ts#L57

GitHub

workers-sdk/packages/wrangler/src/d1/info.ts at 02d40ed3bbfc9cb4c2f...

Home to Wrangler, the CLI for Cloudflare Workers® - cloudflare/workers-sdk

Chaika•5/15/25, 8:39 PM

can see the exact query there. Wrangler's all open source

aza547•5/16/25, 11:36 AM

I'm seeing spikes of D1 failures periodically when querying my database recently. It doesn't feel like the normal level of background failures you might just retry to avoid. The errors all seem to be one of:

D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be resetD1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset
D1_ERROR: Network connection lost.D1_ERROR: Network connection lost.
D1_ERROR: internal error; reference = 3n7qu7dkdd0atgp7vioes6a5D1_ERROR: internal error; reference = 3n7qu7dkdd0atgp7vioes6a5
Is anyone able to advise on this? My database ID is 589bafbd-d600-4e01-8879-7443811d45a1589bafbd-d600-4e01-8879-7443811d45a1 if that's of any use.

garff•5/16/25, 11:39 AM

Hi. I was wondering if it is possible to do transactions using the workers-qb/D1 query bilder? Is there some syntax I do not know about?

Ggarff Hi. I was wondering if it is possible to do transactions using the workers-qb/D1...

gruntlord6•5/17/25, 12:17 AM

Does batch work?

Ggruntlord6 Does batch work?

garff•5/17/25, 11:09 AM

Yea. It greats. Found the right part of the documentation. Thx for replying though!

HHard@Work SQLite DOs are kind of dynamic bindings? They have most of the features, minus r...

Dean•5/18/25, 3:35 PM

Looking into durable objects more today - they need to be binded to a worker which is the same problem I have with d1.

DDean Looking into durable objects more today - they need to be binded to a worker whi...

Hello, I’m Allie!•5/18/25, 4:15 PM

Sort of? You bind a DO namespace to your Worker once, but from that namespace, you pass an ID and get as many DOs as you want

Hello, I’m Allie!•5/18/25, 4:15 PM

One Namespace -> Many “Databases”

gruntlord6•5/19/25, 12:28 AM

yup, you can have a lot of individual objects in a single namespace

gruntlord6•5/19/25, 12:28 AM

for example, an object for every customer

gruntlord6•5/19/25, 12:29 AM

where as with D1 you need a binding for every db

planet•5/19/25, 1:52 AM

Does a primary get replicated to multiple replicas, or just a single replica per region?

e.g. does a WNAM primary gets replicated to a single ENAM location, or multiple ENAM locations?

Pplanet Does a primary get replicated to multiple replicas, or just a single replica per...

Chaika•5/19/25, 2:02 AM

currently just one replica per region, including the primary region

Rripes Bumping this. Any pointers on importing an existing sqlite db that already has a...

Max (@rozenmd)•5/19/25, 8:18 AM

https://developers.cloudflare.com/d1/best-practices/import-export-data/#known-limitations

Disciple of Caoximeihao•5/19/25, 9:16 AM

Is there a way to export the database but still have it read and write normally in production?

Disciple of Caoximeihao•5/19/25, 9:16 AM

Is there a way to export the database but still have it read and write normally in production?

xeon06•5/19/25, 3:43 PM

Is there anyone working on a D1 client for workers with built-in retry, to try and circumvent these issues? https://github.com/cloudflare/cloudflare-docs/issues/18485

xeon06•5/19/25, 3:56 PM

I suppose it isn't that simple to enforce idempotency

.ul0gic•5/19/25, 6:46 PM

anyone here familiar or have experience in getting local D1 instances to maintain persistence during local development by chance?

.ul0gic•5/19/25, 7:35 PM

Fixing D1 Local Persistence with Wrangler Pages Dev

Issue

Local D1 persistence with wrangler pages devwrangler pages dev may not work as expected, even when preview_database_idpreview_database_id is set in

wrangler.toml

wrangler.toml

and --persist-to--persist-to is used in the devdev script. Runtime operations (e.g., API calls creating new data) might connect to a new, blank D1 instance instead of the schema-initialized, persisted one. This leads to "no such table" errors and data loss between development server restarts.

message.txt2.15KB

.ul0gic•5/19/25, 7:37 PM

sorry if that breaks the rules....ive seen a few people ask about this and have this same issue. took multiple days and finally figured it out. hope it helps someone else here struggling with this.

DDean i get DO's exist, but i built the platform around D1 with dynamic bindings comin...

lambrospetrou•5/20/25, 7:39 AM

D1 dynamic bindings are not coming in the short-term, and still undecided for long-term. If you want to programmatically create thousands of databases, SQLite Durable Objects are already available and they are the exact same SQLite components under the hood.

Llambrospetrou D1 dynamic bindings are not coming in the short-term, and still undecided for lo...

Dean•5/20/25, 7:44 AM

It’s a shame that really wasn’t communicated at the start. I’ll have to rewrite everything which is a shame.

Fantastic, thanks for getting to the bottom of it! In the meantime I'll probably try and migrate thi

Issue

Similar Threads

Issue