Cloudflare Developers•5mo ago

~400 queries over the last few hours, all reporting served by primary. Query is a SELECT.

MMatthew ~400 queries over the last few hours, all reporting served by primary. Query is ...

lambrospetrou•8/4/25, 3:03 PM

DM me your database ID and account ID and I can take a look.

Llambrospetrou DM me your database ID and account ID and I can take a look.

lambrospetrou•8/4/25, 3:53 PM

Closing the loop here, the issue was transient during the replication toggle originally when the triggered it not actually persisting the replica enablement. Toggling it on/off again fixed it. We will look into the original error and improve it so that you don't have to toggle it manually.

Abakir•8/4/25, 6:07 PM

This might be a dumb question if a 1000 users try to write to the database at the same time or make a query it won’t fault correct.

I know there are limits of d1 database

AAbakir This might be a dumb question if a 1000 users try to write to the database at th...

James•8/4/25, 6:49 PM

single threaded db, you got 1000ms in a second, gotta work out your usage constraints based on that

Abakir•8/4/25, 6:58 PM

Yes but can multiple people read and write to the database at the same time

AAbakir Yes but can multiple people read and write to the database at the same time

Larry•8/4/25, 7:16 PM

With read replicas multiple can read at the same time, but only one writes at the precise same time. However, as @Ahz was implying, if your definition of "the same time" is in the same second or 100ms, you could have a number of them satisfy that. D1 is built on Durable Objects (DO) and the general understanding that you can have no more than about 1000 requests per second. With write, I suspect that's much lower, but how much would depend upon your writes.

workerBee•8/5/25, 11:04 AM

I've found that it's pretty easy to lock up D1 with concurrent writes, so I add a few ms wait time to all write requests.

WworkerBee I've found that it's pretty easy to lock up D1 with concurrent writes, so I add ...

Larry•8/5/25, 11:10 AM

Can you define “lock up”? Wouldn’t it be better to sense pushback and respond accordingly?

LLarry Can you define “lock up”? Wouldn’t it be better to sense pushback and respond ac...

Larry•8/5/25, 11:15 AM

It should send a 429 too many requests code

workerBee•8/5/25, 11:17 AM

More specifically, if the write requests are originating from a Queue, staggering the requests becomes more important.

Billy•8/5/25, 12:20 PM

I'm trying to simply download my D1 db and I'm seeing a lot of historical conversations about the inability to do that now, is that still the case? Sqlite dbs are just files and I'm hoping to be able to download a read-only version upon request... ideally through an API but I only see the GET for the info, not the actual db. I also don't see a way with wrangler and the time-travel thing anymore now that the "backup" subcommand is gone. I'd prefer not to have to export and then the create another db from that if possible.

BBilly I'm trying to simply download my D1 db and I'm seeing a lot of historical conver...

dandoen•8/5/25, 12:31 PM

have you seen this: https://developers.cloudflare.com/d1/best-practices/import-export-data/#export-an-existing-d1-database and https://developers.cloudflare.com/workflows/examples/backup-d1/ ?

only challenge is that a running export will block other database requests

dandoen•8/5/25, 12:33 PM

per d1-database, the export blocking db requests is getting addressed hopefully this year

Ddandoen have you seen this: https://developers.cloudflare.com/d1/best-practices/import-e...

Billy•8/5/25, 12:36 PM

Thank you, I did see this sadly. I was hoping there was a direct link to the db somehow

Billy•8/5/25, 12:38 PM

this is another option fwiw: https://developers.cloudflare.com/workflows/examples/backup-d1/

Cloudflare Docs

Export and save D1 database

Send invoice when shopping cart is checked out and paid for

BBilly I'm trying to simply download my D1 db and I'm seeing a lot of historical conver...

lambrospetrou•8/5/25, 12:47 PM

You can export your database into an SQL file: https://developers.cloudflare.com/d1/best-practices/import-export-data/#export-an-existing-d1-database As mentioned, while exporting the D1 database is blocked from executing other queries, so this would be the main limitation depending on your requirements.

We do know this falls short in some cases, depending on your database size, and we have plans to improve the export flow. For now, this is the D1 provided way, but as others linked above you can try different libraries doing that too.

LLarry With read replicas multiple can read at the same time, but only one writes at th...

Abakir•8/5/25, 1:12 PM

Will creating a queue for writes be a safe thing to do just in case two writes are trying to happen at the same time?

AAbakir Will creating a queue for writes be a safe thing to do just in case two writes a...

Larry•8/5/25, 1:23 PM

It has some ability to queue built in. When you exceed that, you get back a 429

LLarry It has some ability to queue built in. When you exceed that, you get back a 429

Larry•8/5/25, 1:26 PM

I don't know if D1 does this but other 429 implementations include a header telling you how long to wait before retrying. Inspect the headers in a 429 to confirm if D1 does that… or maybe someone here knows

Abakir•8/5/25, 1:49 PM

Okay will test it out myself and will do

AAbakir Will creating a queue for writes be a safe thing to do just in case two writes a...

lambrospetrou•8/5/25, 2:05 PM

If we step back, what is the main thing you are trying to solve here? Is it the write throughput? How many writes per second do you want to support, and how long does each write take?

Maybe I missed it in the thread above, but adding queues also comes with complexity and their own throughput problems, so want to make sure you are not solving a problem that doesn't exist.

Original question: This might be a dumb question if a 1000 users try to write to the database at the same time or make a query it won’t fault correct.

1000 writes at the same time once will just be serialized, and eventually all go through. Now if you have 1K writes per second consistently, that will almost certainly trigger an overload error to be returned to you and queries starting to be rejected.

Llambrospetrou If we step back, what is the main thing you are trying to solve here? Is it the ...

Abakir•8/5/25, 2:08 PM

It won't happen often for us, but we are trying to prepare if it did so I see it is not a problem for us currently.

However we are scared of the 10GB limit more we might not reach it until after two years, what do you think the best approach to this problem with be with D1 beacause I do not like the idea of scaling databases horizantally, maybe because it does not make that sense for me that each user has it's own database and it feels as managing a users info will be a nightmare that way.

Let me know if I misunderstood anything from what I am saying

AAbakir It won't happen often for us, but we are trying to prepare if it did so I see it...

lambrospetrou•8/5/25, 2:14 PM

If you will be having 1K users writing per second regularly, and you have fear of the 10GB, then I suggest split your database. You don't have to go one DB per user, but maybe split by table if there are obvious ways to split, and have N DBs where you can share the load. If the queries are the same and splitting by table doesn't make sense, then sharding among N DBs works fine too.

It depends how often you will be having this load, and how soon you will hit the storage limit.

Llambrospetrou If you will be having 1K users writing per second regularly, and you have fear o...

Abakir•8/5/25, 2:15 PM

Is there a good example of how to sharding N DBs. I would love to continue to use Cloudflare I just want good tutorials to follow and I can't find many to see how this process feels for larger databases

AAbakir Is there a good example of how to sharding N DBs. I would love to continue to us...

lambrospetrou•8/5/25, 2:17 PM

My usual approach is to pick the identifier that is always accessible for a request (e.g. account ID) and

hash(id) % N

hash(id) % N

to get the DB to use. And then do the normal thing you would do using that DB.

I want to re-iterate though, that if as you say you are 2 years away from this load and hitting the limit, you might as well just leave it for later. But, sharding in general doesn't mean one per user as many folks assume (and some docs/posts say). It's up to you and your data model to find something that works within your needs. If you are going to full-blown one DB per user, using Durable Objects directly is more suited.

Llambrospetrou My usual approach is to pick the identifier that is always accessible for a requ...

Abakir•8/5/25, 2:28 PM

Thank you for this info I just wanted to know this info

Larry•8/5/25, 5:35 PM

we are scared of the 10GB limit more we might not reach it until after two years

IF you approach that limit, you could shard as is being discussed or you could switch to something else. Cloudflare even uses Postgres in places.

Llambrospetrou You can export your database into an SQL file: https://developers.cloudflare.com...

Billy•8/5/25, 7:11 PM

One of the reasons I wanted to use sqlite was the ease of moving a file around, so this is disappointing, but I'm grateful for all that has been done around d1, maybe things will be simpler one day. Thanks for clarifying the plans.

onewug•8/5/25, 10:45 PM

hi! looking at the docs here: https://developers.cloudflare.com/d1/worker-api/prepared-statements/

is it actually true that prepared statements are more performant within a cloudflare workers environment? to me it seems like prepared statements are only more performant if you are performing the same query multiple times within the same connection, since they will be recreated on each connection. or am i misunderstanding something?

Cloudflare Docs

Prepared statement methods

This chapter documents the various ways you can run and retrieve the results of a query after you have prepared your statement.

Oonewug hi! looking at the docs here: https://developers.cloudflare.com/d1/worker-api/pr...

lambrospetrou•8/6/25, 10:17 AM

Prepared statements in D1 are only a client side thing so that you can bind values to the statement. It doesn't change the actual query execution, or duration, at all.

Llambrospetrou Prepared statements in D1 are only a client side thing so that you can bind valu...

onewug•8/6/25, 10:52 AM

then should the docs be updated to not reference performance?

Oonewug then should the docs be updated to not reference performance?

lambrospetrou•8/6/25, 11:29 AM

What that might mean is that if someone just inlines all their data in the SQL itself it might be slower. So, it does have a small benefit in that way. What I meant above is that D1 prepared statements are not like in other databases that you compile a query and avoid overhead by reusing it.
In the current version of D1 since you make a remote call for every query SQLite is given the query SQL string and the params, and does its thing. The prepared statements are not reused across requests at the SQLite engine level.

I will see how we can make that note clearer.

onewug•8/6/25, 1:36 PM

i see, thanks for explaining!

Jun•8/6/25, 2:01 PM

The text has been updated FYI

JJun The text has been updated FYI 😄

Larry•8/6/25, 2:53 PM

That was fast! That's one thing I really love about being a developer on the Cloudflare platform! Thanks Jun!!!

lmsqqt•8/6/25, 4:13 PM

I'm a newcomer. I want to ask if there is a security problem if the ID of the d1 database is directly put on Github?

Llmsqqt I'm a newcomer. I want to ask if there is a security problem if the ID of the d1...

Isaac McFadyen•8/6/25, 4:21 PM

No security issue, it's fine. All binding IDs can be posted in public settings, they only work on your own account.

lmsqqt•8/6/25, 4:28 PM

thank

ryan•8/7/25, 2:08 AM

CeeJay•8/7/25, 9:59 AM

I don't think Cloudflare did a great job choosing the region for my D1, I should have put a location hint in (if that was available when I created it?).

Anyhow, now I am in a position that the DB (and likely the workers) are resident in Western North America and all my workload originates in UK.

Apart from exporting the whole thing, importing into a Western Europe resident DB are there any tools or patterns for transitioning and ensuring consistency / syncing?

TIA

CCeeJay I don't think Cloudflare did a great job choosing the region for my D1, I should...

lambrospetrou•8/7/25, 10:51 AM

Unfortunately, we don't provide any tooling to transfer databases at the moment, and we cannot move them after creation. I suggest creating a new DB providing a location hint to ~~EEUR~~ WEUR (includes UK), and copy your data there.

Llambrospetrou Unfortunately, we don't provide any tooling to transfer databases at the moment,...

CeeJay•8/7/25, 10:54 AM

Thanks for coming back to me

- Can you confirm that UK is in/closest to Eastern Europe, it's usually categorised as Western Europe?

CCeeJay Thanks for coming back to me 🙏 - Can you confirm that UK is in/closest to Easte...

lambrospetrou•8/7/25, 10:58 AM

Yeah sorry, I meant WEUR, see also https://where.durableobjects.live/colo/LHR to see latencies across all location hints.
Your DB will end up in the UK or Amsterdam probably but depending on datacenter issues might move into other WEUR locations.

Where Durable Objects Live

London, England, United Kingdom (colo)

Tracking where Durable Objects are created, wherever you are in the world.

Billy•8/7/25, 8:27 PM

In this doc: https://developers.cloudflare.com/d1/best-practices/import-export-data/#known-limitations, the statement "A running export will block other database requests." really only applies to

CREATE

CREATE

INSERT

INSERT

, and

DELETE

DELETE

requests, correct? If so, it would've helped assure me that

READ

READ

would not be blocked during an export (if that's true). Thanks

Cloudflare Docs

Import and export data

D1 allows you to import existing SQLite tables and their data directly, enabling you to migrate existing data into D1 quickly and easily. This can be useful when migrating applications to use Workers and D1, or when you want to prototype a schema locally before importing it to your D1 database(s).

gruntlord6•8/7/25, 8:32 PM

reads are blocked

BBilly In this doc: https://developers.cloudflare.com/d1/best-practices/import-export-...

gruntlord6•8/7/25, 8:32 PM

the whole thing is not accessible while its dumping the database

Ggruntlord6 reads are blocked

Billy•8/7/25, 9:10 PM

yikes, ok... but that blocking limitation is about to be eliminated, correct?

gruntlord6•8/7/25, 9:15 PM

I mean they want to eliminate it, they haven't given a timeline of any kind. You can always just copy the data directly instead of using that tool

ldo•8/8/25, 8:32 AM

Is there a page or dashboard where I can check the server health/status of D1? One of my D1 services has been returning timeouts or 500 errors for about two hours, and I would like to know when this issue is expected to be resolved.

Lldo Is there a page or dashboard where I can check the server health/status of D1? O...

lambrospetrou•8/8/25, 10:00 AM

There isn't any widespread D1 service issue or individual DBs with startup issues. If you continue having 5xx please send us more details. DB/Account IDs, and any timeranges you can specify to help us investigate.

~400 queries over the last few hours, all reporting served by primary. Query is a SELECT.

Similar Threads

Similar Threads

Similar Threads