Hi everyone, I have a 3gb database with multiple keys (one line can be quite large - a few kb - but

Hi everyone, I have a 3gb database with multiple keys (one line can be quite large - a few kb - but the blobs are not part of any keys), the maximum size is 10gb, but for some reason querying even this small DB is very slow (even simple queries take up to a minute). I use composite keys. Anybody else seeing this type of behavior? Noteworthy is the fact that this db experiences a lot of writes as it is a live stats marting table.

Spencer•5/28/25, 2:39 PM

hey, i need some help please, got an error D1 ERROR: Currently processing a long-running export, how to cancel this operation ? cause because of it my app stops working

SSpencer hey, i need some help please, got an error D1 ERROR: Currently processing a long...

lambrospetrou•5/28/25, 3:40 PM

If you are doing an export the whole DB is blocked until it uploads everything to R2. If you want to abort the export, then unfortunately you need to stop polling and wait a few seconds for it to stop.

PPaul Mihai Hi everyone, I have a 3gb database with multiple keys (one line can be quite lar...

lambrospetrou•5/28/25, 3:42 PM

What is the

.meta

.meta

information telling you about the duration of the query itself? 1 minute is very long either way and probably hints to other issues, commonly overloaded DB due to traffic or slow-ish consecutive queries.
Also, are all your queries slow, or just that specific one?

Paul MihaiOP•5/28/25, 3:42 PM

Writing seems to be perfectly fine, but querying especially on high writing load is suuuuper slow

Paul MihaiOP•5/28/25, 3:43 PM

we have a few million writes / day

PPaul Mihai Writing seems to be perfectly fine, but querying especially on high writing load...

lambrospetrou•5/28/25, 3:43 PM

Keep in mind that D1 is single threaded. Only one query executes at a time.

Paul MihaiOP•5/28/25, 3:43 PM

but just a few reads a day

Paul MihaiOP•5/28/25, 3:43 PM

so the problem may be the queue

Paul MihaiOP•5/28/25, 3:43 PM

because we have so many writes?

PPaul Mihai so the problem may be the queue

lambrospetrou•5/28/25, 3:44 PM

Yes, maybe. Check the

meta

meta

field of the responses for the duration listed. If that's small (a few milliseconds) then the issue is not the query itself. It can be either the volume of your load (number of requests and time each write query takes to finish) or some networking issue. The latency of the query is also available in the UI and through wrangler insights.

Paul MihaiOP•5/28/25, 3:45 PM

I'll do that! Thanks! Also I understood there is a "new" version of D1?

Paul MihaiOP•5/28/25, 3:46 PM

is that still a thing or were all migrated to the new backend?

Llambrospetrou If you are doing an export the whole DB is blocked until it uploads everything t...

Spencer•5/28/25, 3:46 PM

but i am not polling atm, it's just stuck, there is a way to cancel the process somehow ?

PPaul Mihai I'll do that! Thanks! Also I understood there is a "new" version of D1?

lambrospetrou•5/29/25, 10:30 AM

You probably mean a new version of Durable Objects, not D1. The SQLite based DOs are new-ish yes, but D1 was already built ontop of those. There is no new version of D1.

Zig•5/29/25, 6:23 PM

Does anyone have experience troubleshooting

D1_ERROR: Failed to parse body as JSON, got: Error: internal error;

D1_ERROR: Failed to parse body as JSON, got: Error: internal error;

D1_ERROR: database is locked: SQLITE_BUSY

D1_ERROR: database is locked: SQLITE_BUSY

in local development? im hoping this goes away when i deploy it...

Zig•5/29/25, 6:36 PM

Im running a pages app with D1 locally, and a DO thats accessed through both a DO binding in pages and a separate worker for the DO thats sometimes called by pages. The DO worker is ran with wrangler persisting the state to the pages worker D1 instance. The pages worker calls the DO to do some work and it modifies D1, the DO sends web socket messages to pages occasionally as it runs forcing pages to fetch D1 data.

I am getting seemingly random SQLITE and D1 errors that appear in random order, and arent confined to any one section of my code. Is this just a byproduct of trying to use 2 instances of wrangler, one with the persist-to flag on local environments? Or is this indicative of a problem with my usage patterns? Will this still happen on deployed instances?

Zig•5/29/25, 9:45 PM

Update: Deployed version works flawlessly. Guessing its some issue with 2 wrangler instances trying to use the D1 file at the same time

lambrospetrou•5/30/25, 3:19 PM

We released some improvements to the D1 REST API a few days ago and now most endpoints should have some speedup. The query endpoints especially now should be much faster if you are outside the US.

Read more in the changelog: https://developers.cloudflare.com/changelog/2025-05-30-d1-rest-api-latency/

Would love to see if anyone notices this in their graphs.

Cloudflare Docs

50-500ms Faster D1 REST API Requests

Llambrospetrou 🚀 We released some improvements to the D1 REST API a few days ago and now most...

Chaika•5/30/25, 3:42 PM

Is it still subject to the normal API Rate Limits? (1200/5m)?

CChaika Is it still subject to the normal API Rate Limits? (1200/5m)?

lambrospetrou•5/30/25, 3:56 PM

No changes were done there, yes.

Llambrospetrou No changes were done there, yes.

Chaika•5/30/25, 4:23 PM

aw, that's a pretty big blocker for using the API for Dynamic D1 stuff, hopefully in the future it will be adjusted

CChaika aw, that's a pretty big blocker for using the API for Dynamic D1 stuff, hopefull...

lambrospetrou•5/30/25, 4:25 PM

Yeah there is a bit more work to do there, but if you have high traffic going through a worker is still best. REST API has the rate limiting, still needs to do auth, check your plan status, etc, so it will always have some delay even if rate limiting was higher. But yes, rate limiting is a common issue with the REST API. It's a thing folks want to improve across CF overall.

Muhammad nisar 0372•6/1/25, 6:32 AM

Muhammad nisar 0372•6/1/25, 6:33 AM

@lambrospetrou @lambrospetrou

Joe•6/3/25, 10:44 AM

Hello! I'm strugling to create a new D1 database (via wrangler and the web dashboard). I get an error about not being able to create more than 10, despite being a paid user with a limit of 50,000.

Any ideas?

JJoe Hello! I'm strugling to create a new D1 database (via wrangler and the web dashb...

Brayden•6/3/25, 2:29 PM

Are you still experiencing a problem here? Can you DM me your account ID?

Melissa1181•6/4/25, 5:51 AM

Hey

Rripes Claude integrations support was just released on Pro plan, and I really want to ...

lambrospetrou•6/4/25, 6:22 AM

Not at the moment, unfortunately, you will have to make sure all queries going through are read only.

Rripes Claude integrations support was just released on Pro plan, and I really want to ...

gruntlord6•6/5/25, 2:38 PM

could you have a worker in the middle that prevents writes from being passed?

gruntlord6•6/5/25, 2:38 PM

alternatively you could download the d1 export and spin up a local sqlite database

dsds10•6/5/25, 7:57 PM

Regarding the 50k db (soft) limit, how likely is this limit to be increased for non-enterprise customers?

davidbarratt•6/7/25, 12:55 AM

can you dynamically bind a D1 database? I'd like to have a database per-tennat that gets selected based on the request, but it doesn't look like that is posible without just using the REST API directly? Is that right?

Ddavidbarratt can you dynamically bind a D1 database? I'd like to have a database per-tennat t...

1984 Ford Laser•6/7/25, 12:57 AM

Durable Objects will be your best solution

11984 Ford Laser Durable Objects will be your best solution

davidbarratt•6/7/25, 1:02 AM

TIL about the SQL storage of Durable Objects... thanks!

Ddavidbarratt TIL about the SQL storage of Durable Objects... thanks!

1984 Ford Laser•6/7/25, 1:02 AM

You'll probably need a bit more boilerplate than with D1 but its a far better solution overall

Brayden•6/7/25, 1:23 AM

https://github.com/outerbase/starbasedb <- A project I made before joining Cloudflare but helps with the boilerplate on top of DO's for a D1 like experience.

BBrayden https://github.com/outerbase/starbasedb <- A project I made before joining Cloud...

dsds10•6/7/25, 8:44 AM

hmm that's interesting. somehow I had the "sqlite in do" stored as "ephemeral" in my mind, but checking the docs that's not the case. I also read that the max size in DO was 1GB, but this doesn't seem the case anymore (or never was)

what are the pro/cons of using sqlite in DO vs using D1?

bkyerv•6/7/25, 9:01 AM

why does wrangler d1 create [name] not work with oauth token (results in auth error) and requires api token? is it by design? or is it a known bug?

Ddsds10 hmm that's interesting. somehow I had the "sqlite in do" stored as "ephemeral" i...

Brayden•6/7/25, 11:04 AM

D1 is actually built on top of DO's as an interesting note. D1 uses the SQLite storage layer of DO's but as part of D1 there are also a bunch of boilerplate and helper functionality built on top to make it easier to interact with.

I also read that the max size in DO was 1GB

They did use to be 1GB not too long ago, but not it is 10GB

what are the pro/cons of using sqlite in DO vs using D1

Pros of DO: you have a lot more flexibility to do what you want. Compute & storage all at the same time. Cons of DO: you have to build the boilerplate interfacing systems to inspect them at the moment (but there are libraries that do this). From a storage layer perspective.. they are both DO SQLite!

Ddsds10 hmm that's interesting. somehow I had the "sqlite in do" stored as "ephemeral" i...

Larry•6/7/25, 11:12 AM

Pros: dynamic binding, hibernating WebSockets, ACID with stored procedures written in JavaScript

Cons: mental model shift from traditional 3 tier architecure

LLarry Pros: dynamic binding, hibernating WebSockets, ACID with stored procedures writt...

1984 Ford Laser•6/7/25, 11:56 AM

That mental model shift can be really hard for some people

dsds10•6/7/25, 1:26 PM

@Brayden thanks! what libraries are you referring to? Just made some tests to auto create instances of my backend system on demand and it works surprisingly well! kinda feels like making a cloud self service in a day

Am I right to assume read/writes are not counted like in D1 as it's effectively querying it "locally"? I think just the requests to the DO and the duration it lives is billed afaik.

@Larry seems like everything has it's pro's and con's

only scratched the surface of DO's so far, but it works great

dsds10•6/7/25, 2:37 PM

works like a charm! in case anyone curious, here is an example: https://example.bknd.run/ (ACL enabled)
you can create your own instance by changing the subdomain to whatever you like

really cool what D1 and DO enables us to create :cloudflare:

thipperz•6/7/25, 3:55 PM

My D1 is dead has been 10 hours

Apparently this happens if you try to export a remote database that has fts5 tables on it.

I would appreciate if someone from Cloudflare could take a look at workers-and-pages-helpD1 ERROR: Currently processing a long-running export

Hi everyone, I have a 3gb database with multiple keys (one line can be quite large - a few kb - but

Similar Threads

Similar Threads

Similar Threads