Cloudflare Developers•5mo ago

Will creating a queue for writes be a safe thing to do just in case two writes are trying to happen

Will creating a queue for writes be a safe thing to do just in case two writes are trying to happen at the same time?

AAbakir Will creating a queue for writes be a safe thing to do just in case two writes a...

Larry•8/5/25, 1:23 PM

It has some ability to queue built in. When you exceed that, you get back a 429

LLarry It has some ability to queue built in. When you exceed that, you get back a 429

Larry•8/5/25, 1:26 PM

I don't know if D1 does this but other 429 implementations include a header telling you how long to wait before retrying. Inspect the headers in a 429 to confirm if D1 does that… or maybe someone here knows

AbakirOP•8/5/25, 1:49 PM

Okay will test it out myself and will do

AAbakir Will creating a queue for writes be a safe thing to do just in case two writes a...

lambrospetrou•8/5/25, 2:05 PM

If we step back, what is the main thing you are trying to solve here? Is it the write throughput? How many writes per second do you want to support, and how long does each write take?

Maybe I missed it in the thread above, but adding queues also comes with complexity and their own throughput problems, so want to make sure you are not solving a problem that doesn't exist.

Original question: This might be a dumb question if a 1000 users try to write to the database at the same time or make a query it won’t fault correct.

1000 writes at the same time once will just be serialized, and eventually all go through. Now if you have 1K writes per second consistently, that will almost certainly trigger an overload error to be returned to you and queries starting to be rejected.

Llambrospetrou If we step back, what is the main thing you are trying to solve here? Is it the ...

AbakirOP•8/5/25, 2:08 PM

It won't happen often for us, but we are trying to prepare if it did so I see it is not a problem for us currently.

However we are scared of the 10GB limit more we might not reach it until after two years, what do you think the best approach to this problem with be with D1 beacause I do not like the idea of scaling databases horizantally, maybe because it does not make that sense for me that each user has it's own database and it feels as managing a users info will be a nightmare that way.

Let me know if I misunderstood anything from what I am saying

AAbakir It won't happen often for us, but we are trying to prepare if it did so I see it...

lambrospetrou•8/5/25, 2:14 PM

If you will be having 1K users writing per second regularly, and you have fear of the 10GB, then I suggest split your database. You don't have to go one DB per user, but maybe split by table if there are obvious ways to split, and have N DBs where you can share the load. If the queries are the same and splitting by table doesn't make sense, then sharding among N DBs works fine too.

It depends how often you will be having this load, and how soon you will hit the storage limit.

Llambrospetrou If you will be having 1K users writing per second regularly, and you have fear o...

AbakirOP•8/5/25, 2:15 PM

Is there a good example of how to sharding N DBs. I would love to continue to use Cloudflare I just want good tutorials to follow and I can't find many to see how this process feels for larger databases

AAbakir Is there a good example of how to sharding N DBs. I would love to continue to us...

lambrospetrou•8/5/25, 2:17 PM

My usual approach is to pick the identifier that is always accessible for a request (e.g. account ID) and hash(id) % Nhash(id) % N to get the DB to use. And then do the normal thing you would do using that DB.

I want to re-iterate though, that if as you say you are 2 years away from this load and hitting the limit, you might as well just leave it for later. But, sharding in general doesn't mean one per user as many folks assume (and some docs/posts say). It's up to you and your data model to find something that works within your needs. If you are going to full-blown one DB per user, using Durable Objects directly is more suited.

Llambrospetrou My usual approach is to pick the identifier that is always accessible for a requ...

AbakirOP•8/5/25, 2:28 PM

Thank you for this info I just wanted to know this info

Larry•8/5/25, 5:35 PM

we are scared of the 10GB limit more we might not reach it until after two years

IF you approach that limit, you could shard as is being discussed or you could switch to something else. Cloudflare even uses Postgres in places.

Llambrospetrou You can export your database into an SQL file: https://developers.cloudflare.com...

Billy•8/5/25, 7:11 PM

One of the reasons I wanted to use sqlite was the ease of moving a file around, so this is disappointing, but I'm grateful for all that has been done around d1, maybe things will be simpler one day. Thanks for clarifying the plans.

onewug•8/5/25, 10:45 PM

hi! looking at the docs here: https://developers.cloudflare.com/d1/worker-api/prepared-statements/

is it actually true that prepared statements are more performant within a cloudflare workers environment? to me it seems like prepared statements are only more performant if you are performing the same query multiple times within the same connection, since they will be recreated on each connection. or am i misunderstanding something?

Cloudflare Docs

Prepared statement methods

This chapter documents the various ways you can run and retrieve the results of a query after you have prepared your statement.

Oonewug hi! looking at the docs here: https://developers.cloudflare.com/d1/worker-api/pr...

lambrospetrou•8/6/25, 10:17 AM

Prepared statements in D1 are only a client side thing so that you can bind values to the statement. It doesn't change the actual query execution, or duration, at all.

Llambrospetrou Prepared statements in D1 are only a client side thing so that you can bind valu...

onewug•8/6/25, 10:52 AM

then should the docs be updated to not reference performance?

Oonewug then should the docs be updated to not reference performance?

lambrospetrou•8/6/25, 11:29 AM

What that might mean is that if someone just inlines all their data in the SQL itself it might be slower. So, it does have a small benefit in that way. What I meant above is that D1 prepared statements are not like in other databases that you compile a query and avoid overhead by reusing it.
In the current version of D1 since you make a remote call for every query SQLite is given the query SQL string and the params, and does its thing. The prepared statements are not reused across requests at the SQLite engine level.

I will see how we can make that note clearer.

onewug•8/6/25, 1:36 PM

i see, thanks for explaining!

Jun•8/6/25, 2:01 PM

The text has been updated FYI

JJun The text has been updated FYI 😄

Larry•8/6/25, 2:53 PM

That was fast! That's one thing I really love about being a developer on the Cloudflare platform! Thanks Jun!!!

lmsqqt•8/6/25, 4:13 PM

I'm a newcomer. I want to ask if there is a security problem if the ID of the d1 database is directly put on Github?

Llmsqqt I'm a newcomer. I want to ask if there is a security problem if the ID of the d1...

Isaac McFadyen•8/6/25, 4:21 PM

No security issue, it's fine. All binding IDs can be posted in public settings, they only work on your own account.

lmsqqt•8/6/25, 4:28 PM

thank

ryan•8/7/25, 2:08 AM

CeeJay•8/7/25, 9:59 AM

I don't think Cloudflare did a great job choosing the region for my D1, I should have put a location hint in (if that was available when I created it?).

Anyhow, now I am in a position that the DB (and likely the workers) are resident in Western North America and all my workload originates in UK.

Apart from exporting the whole thing, importing into a Western Europe resident DB are there any tools or patterns for transitioning and ensuring consistency / syncing?

TIA

CCeeJay I don't think Cloudflare did a great job choosing the region for my D1, I should...

lambrospetrou•8/7/25, 10:51 AM

Unfortunately, we don't provide any tooling to transfer databases at the moment, and we cannot move them after creation. I suggest creating a new DB providing a location hint to ~~EEUR~~ WEUR (includes UK), and copy your data there.

Llambrospetrou Unfortunately, we don't provide any tooling to transfer databases at the moment,...

CeeJay•8/7/25, 10:54 AM

Thanks for coming back to me

- Can you confirm that UK is in/closest to Eastern Europe, it's usually categorised as Western Europe?

CCeeJay Thanks for coming back to me 🙏 - Can you confirm that UK is in/closest to Easte...

lambrospetrou•8/7/25, 10:58 AM

Yeah sorry, I meant WEUR, see also https://where.durableobjects.live/colo/LHR to see latencies across all location hints.
Your DB will end up in the UK or Amsterdam probably but depending on datacenter issues might move into other WEUR locations.

Where Durable Objects Live

London, England, United Kingdom (colo)

Tracking where Durable Objects are created, wherever you are in the world.

Billy•8/7/25, 8:27 PM

In this doc: https://developers.cloudflare.com/d1/best-practices/import-export-data/#known-limitations, the statement "A running export will block other database requests." really only applies to CREATECREATE, INSERTINSERT, and DELETEDELETE requests, correct? If so, it would've helped assure me that READREAD would not be blocked during an export (if that's true). Thanks

Cloudflare Docs

Import and export data

D1 allows you to import existing SQLite tables and their data directly, enabling you to migrate existing data into D1 quickly and easily. This can be useful when migrating applications to use Workers and D1, or when you want to prototype a schema locally before importing it to your D1 database(s).

gruntlord6•8/7/25, 8:32 PM

reads are blocked

BBilly In this doc: https://developers.cloudflare.com/d1/best-practices/import-export-...

gruntlord6•8/7/25, 8:32 PM

the whole thing is not accessible while its dumping the database

Ggruntlord6 reads are blocked

Billy•8/7/25, 9:10 PM

yikes, ok... but that blocking limitation is about to be eliminated, correct?

gruntlord6•8/7/25, 9:15 PM

I mean they want to eliminate it, they haven't given a timeline of any kind. You can always just copy the data directly instead of using that tool

ldo•8/8/25, 8:32 AM

Is there a page or dashboard where I can check the server health/status of D1? One of my D1 services has been returning timeouts or 500 errors for about two hours, and I would like to know when this issue is expected to be resolved.

Lldo Is there a page or dashboard where I can check the server health/status of D1? O...

lambrospetrou•8/8/25, 10:00 AM

There isn't any widespread D1 service issue or individual DBs with startup issues. If you continue having 5xx please send us more details. DB/Account IDs, and any timeranges you can specify to help us investigate.

Llambrospetrou There isn't any widespread D1 service issue or individual DBs with startup issue...

ldo•8/8/25, 10:21 AM

It was not a 100% failure, but the connection was very unstable.
The UUID of the problematic D1 is e6cdb51a-42e0-4a40-a342-bf1852204a87, and the following issues occurred:

D1_ERROR: Network connection lost.
D1_ERROR: D1 DB is overloaded. Requests queued for too long.
Timeout (30s)
When attempting to access the DB via the console, it would load indefinitely.

Currently, the situation has somewhat stabilized and console access is possible.
However, timeouts or response times exceeding 10 seconds still remain.

Lldo It was not a 100% failure, but the connection was very unstable. The UUID of the...

lambrospetrou•8/8/25, 11:10 AM

OK, these errors are not from the D1 service itself, so that's good at least

Network connection lost errors are indeed internal errors from the runtime, usually due to the underlying Durable Object connectivity issues. Some amount of these is to be expected but should be very transient and not ongoing, feel free to retry these if your query allows. If you see these continuously please let me/us know, since sometimes a database might be in a faulty location for a few minutes before it gets taken out of the network or might need something manual on our side.

The overloaded error indicates that either you send too many requests and they are slow being processed, so the queue is growing and at some point extra reqs will be rejected with that error.

I will take a glance on your DB, if you DM me your account ID too.

lambrospetrou•8/8/25, 11:12 AM

As a side note to everyone, it makes it faster for us to investigate issues in your accounts if you connect the account info to your Discord profile, otherwise we would need to ask you all the time about it, meaning more back and forth comms before we can do anything.

Run /link/link and fill the info, so that we can figure out your account by your profile.

JJun The text has been updated FYI 😄

cortopy•8/8/25, 5:48 PM

This is so much clearer!! Would it be possible to maybe clarify the same for durable objects sql api? I’ve been trying to figure out if sql methods protect against sql injection

https://developers.cloudflare.com/durable-objects/api/storage-api/#returns

Cloudflare Docs

Durable Object Storage

The Durable Object Storage API allows Durable Objects to access transactional and strongly consistent storage. A Durable Object's attached storage is private to its unique instance and cannot be accessed by other objects.

Llambrospetrou As long as you use the D1 Binding from your worker, or Pages function, it should...

Dean•8/11/25, 6:19 AM

i enabled it and haven't been able to have a single instance where it's used the replicated db. served_by_region: "WNAM",served_by_primary: true. does it take a few days for it to start serving from different regions?

DDean i enabled it and haven't been able to have a single instance where it's used the...

lambrospetrou•8/11/25, 6:39 AM

No, creating the replicas is immediate, although the couple first requests per location will go to the primary until the replica boots. The most common case of not using replicas is when the request rate is not high enough, for example if you do a request every few minutes in each region then most likely all of them will go to the primary.

We know a small bug on our side for the "enabling" itself which we plan on fixing soon, so if you send constant traffic to your DB and still get no replicas, try disabling it and re-enabling it, and if you still get no replicas DM me your details to take a look.

Llambrospetrou No, creating the replicas is immediate, although the couple first requests per l...

Dean•8/11/25, 6:41 AM

thx - let me try that and if not ill send you db ids. edit disable/reenable fixed it.

can•8/11/25, 10:05 AM

Hello y'all, i ran d1 exportd1 export for one of my databases and now it says Error in worker: Error: D1_ERROR: Currently processing a long-running export.Error in worker: Error: D1_ERROR: Currently processing a long-running export. and i'm not sure it's currently processing because it errored out on my terminal. Is there a way to cancel this process?

can•8/11/25, 10:11 AM

how am i suppose to stop polling for that process?

ToonyGen•8/11/25, 7:09 PM

I just found a fancy repo to manage d1. https://github.com/earth-app/CollegeDB.git

GitHub

GitHub - earth-app/CollegeDB: Cloudflare D1 Sharding Router for Hor...

Cloudflare D1 Sharding Router for Horizontal Scaling - earth-app/CollegeDB

red2green•8/12/25, 3:05 AM

are there any case stuides/examples of cloudflare customers that have thousands of dbs? Curious what its like to actually have to manage that many databases in production.

TToonyGen I just found a fancy repo to manage d1. https://github.com/earth-app/CollegeDB.g...

lambrospetrou•8/12/25, 9:41 AM

Really cool. I like the simplicity of using an explicitly provided "shard/hash key". It makes the routing much simpler. This is similar approach to how I shard across SQLite DOs.

I wonder how this was developed, was it a work project, college project (hint in the name), or just a side project? Majority of the logic seems to have been added in a single commit.

Either way, very nice to read through. Haven't checked the code yet. This is not an endorsement to use this in production, BTW.

Rred2green are there any case stuides/examples of cloudflare customers that have thousands ...

lambrospetrou•8/12/25, 9:44 AM

Unfortunately, I am not aware of publicly documented cases. We do have a few users that are like agencies developing apps for other companies and use thousands of D1 DBs. They mostly manage them through the REST API. Most of these cases are not attaching thousands of DBs to a single worker, as far as I know, although some did and use the Workers Patch API to update the bindings on their worker dynamically.

My suggestion, if you have a single worker project and want thousands of DBs using SQLite DOs will give you a much nicer experience. Now, if you are a platform creating thousands of projects and each one needs a small D1 DB that's a great use-case for D1 and managing the DBs through the API.

Llambrospetrou Unfortunately, I am not aware of publicly documented cases. We do have a few use...

Levi•8/12/25, 12:25 PM

I essentially came here to ask about dynamically creating and binding of D1 databases, but it seems like using Durable Objects would work better.

My goal is to have a database per user, org, or tenant for my one app.

Llambrospetrou Really cool. I like the simplicity of using an explicitly provided "shard/hash k...

ToonyGen•8/12/25, 1:25 PM

I think the author is using as the backend for its own social media project.

LLevi I essentially came here to ask about dynamically creating and binding of D1 data...

cloudkite•8/12/25, 8:27 PM

We are building a db per customer architecture and were holding out for dynamic bindings. However decided to bite the bullet and currently migrating from d1 to durable objects. Was initially reluctant but have to say it gives you a lot more flexibility and as a bonus has opened up some nice opportunities for performance optimisations.

Ccloudkite We are building a db per customer architecture and were holding out for dynamic ...

red2green•8/12/25, 8:43 PM

with a durable object, backed by sqlite, does it act just like a sqlite db? you can create many tables etc? DO is a little confusing.

Will creating a queue for writes be a safe thing to do just in case two writes are trying to happen

Similar Threads

Similar Threads

Similar Threads