Before I test, how many rows per second can D1 / DO with SQL scan per second? any unofficial data ar
Before I test, how many rows per second can D1 / DO with SQL scan per second? any unofficial data around it?
LI have seen high hundreds of thousands or single digit millions reads per second. Obviously it's different if your rows are 10 bytes vs 2MB!
Also, with such high number of reads per second, if you are doing that constantly you need to be careful of hitting CPU limits, or overload limits since a DO is single threaded.
Also, with such high number of reads per second, if you are doing that constantly you need to be careful of hitting CPU limits, or overload limits since a DO is single threaded.
CHi, I have a question about D1. I'm using it together with Drizzle ORM. I have a table that stores user balances, and each request deducts from the balance. The issue is that a simple query like
amount -= 100
LI have seen high hundreds of thousands or single digit millions reads per second.Wait! How is that possible? I've been using no more 1,000 request/messages per second as design ceiling for DOs, eventhough, I've seen as high as 5,000. Even assuming 5,000, you'd need 200 read replicas to hit 1M. Does D1 go to that level?
T@Larry if think we are discussing different things, I was asking how many rows per second can be scanned, so if you have a table with 100k rows, how long would it take to scan it, vs request per second which would be how many queries can run in a second
LDuh, I should read more carefully. Thanks for the clarification! That makes a lot more sense.
TNo problems
CWould an atomic update be sufficient?
Or using durable objects.
Or using durable objects.
JI like using DOs for transactions more simply because you control the thread of processing. Run a bunch of SQL in synchronous code with the right error handling in place and you have transactions.
Another approach I use when coordinating changes async across D1/DO other resources in a saga pattern.
Another approach I use when coordinating changes async across D1/DO other resources in a saga pattern.
BI saw the post about the rollout of data jurisdictions. Love it! Will it be possible to enable retroactively for existing DBs? I've got a project in staging that's about to go live and I really don't want to recreate the DB
ZThere should definitely be something like development branches for D1. This would be a great addition as it would compliment preview deployments from git repositories. That way, developers could test database changes in a staging enviroment without modifying the prod database.
LIf you can build all the queries you want to run in a transaction before hand, you can send them all in a single
So, for example, if you want to update two tables, or two different rows but with separate queries, you can prepare both of them and send them together.
https://developers.cloudflare.com/d1/worker-api/d1-database/#batch
batch()So, for example, if you want to update two tables, or two different rows but with separate queries, you can prepare both of them and send them together.
https://developers.cloudflare.com/d1/worker-api/d1-database/#batch
Cloudflare Docs
To interact with your D1 database from your Worker, you need to access it through the environment bindings provided to the Worker (env).
CThis is great — but could it still lead to a race condition issue?
LYou can always add race conditions if you split something that should be a single transaction across multiple requests.
I am just mentioning that there is the
If you need to intervleave JS code with SQL queries, then D1 cannot do that, and your best bet would be SQLite DOs.
I am just mentioning that there is the
batch()If you need to intervleave JS code with SQL queries, then D1 cannot do that, and your best bet would be SQLite DOs.
CI got it, thanks.
MIn both the CF dash and via the D1 REST API's list DBs endpoint, it says my database has zero tables. This isn't true; it has 2, and I can successfully query data. These tables have existed for several days. Why would this be happening?
LI believe the GET database details should give you the correct information.
The list endpoint I think returns stale data due to a control plane change few weeks ago. I will report it internally to be fixed.
The list endpoint I think returns stale data due to a control plane change few weeks ago. I will report it internally to be fixed.
TAbout database scaling limits, is there a way I can check my current usage?
LThe graphql API returns rows read/written and db sizes per database. The dashboard also shows that data in the metrics tab.
JIm curious (and forgive my stupidity here) if there's an officiall or unofficially blessed way to do D1 multitenancy that allows for using parameters in queries. Everything I'm seeing seems to indicate people are doing like hash sharding and stuff to allow them to leverage worker bindings when they scale a per tenant or per entity D1 architechture. Im trying to do a rewrite to pivot that direction (multitenancy) but you can't bind parameters to multi statement queries over http (im not even sure you could do that with bindings I've never tried). And it seems the efficiency and performance hit of doing an independant api call for each query would be pretty bad... Should I just be abandoning parameters and doing my own sanitization so I can batch calls over the API with all my values written into it? or am missing a "blessed" pathway here?
JIt just seems like hacky "redeploys" where I eternally bind more D1 databases to my wrangler.json can't be the best way to go about this... but also I don't want to run unsafe SQL statements and I don't want to run an api call for each query... If there's no official way I'll figure something out but it seems like I must be missing something here.
Alternatively, is this something I should be solving with Queues?
Alternatively, is this something I should be solving with Queues?
LIf you want thousands of DBs (tenants) you should use SQLite Durable Objects. Same underlying SQLite engine as D1, but you can dynamically create thousands of instances programmatically.
LHey folks, we updated the section "How much work can a D1 database do?" in our docs: https://developers.cloudflare.com/d1/platform/limits/#how-much-work-can-a-d1-database-do
It tries to give a guidance on what to expect, and set expectations a bit more realisitically. Let me know how you find it.
It tries to give a guidance on what to expect, and set expectations a bit more realisitically. Let me know how you find it.
Cloudflare Docs
Cloudflare also offers other storage solutions such as Workers KV, Durable Objects, and R2. Each product has different advantages and limits. Refer to Choose a data or storage product to review which storage option is right for your use case.
Jlooks like I have some reading to do thanks 
LCloudflare Docs
A Durable Object is a special kind of Cloudflare Worker which uniquely combines compute with storage. Like a Worker, a Durable Object is automatically provisioned geographically close to where it is first requested, starts up quickly when needed, and shuts down when idle. You can have millions of them around the world. However, unlike regular Wo...
TAre there any plans to add more read replicas locations? i.e. South America
C(which aren't in South America yet)
Cnot been any new DO regions for a bit, middle of last year they said they were trying to get India and then South America by end of 2024...
durable-objects
durable-objects
Aanyway to connect d1 with a db GUI. Looked through docs and I think it is not possible. just wanted to make sure I have not missed a work around or something.
Thanks
Thanks
1The first native studio for D1 databases
ADoes this let me pin queries? Cloudflare D1 also has a GUI but it's limited. I have some long admin queries I wanna keep around the db instead of my notepad
GYou need to optimize your query, whatever you are doing is more then 128mb which is how much in memory state the d1 durable object has.
Have you tried splitting the reads into paginated requests rather then just selecting all 26 million rows?
Have you tried splitting the reads into paginated requests rather then just selecting all 26 million rows?
1You definitely wanna add the index before having 26mil rows. I would also consider if there are ways to split the data up into separate tables, or only select a subset at a time
LAs others suggested, this is one of the most common ways folks hit the SQLite memory issues with D1 (and Durable Objects). Doing aggregations across the whole table of millions of rows is too easy to hit the memory/CPU limits.
In your specific query:
Although, that would need an index to work properly if you don't already have one, but it's worth trying.
In your specific query:
SELECT calc_date, indicator_id FROM GridData GROUP BY calc_date, indicator_idAlthough, that would need an index to work properly if you don't already have one, but it's worth trying.
KIsn't hyperdrive and option for you?
LHey folks, a quick update for those that enabled read replication in the past few weeks but didn't see replicas appearing. There was a control plane bug that in some edge cases even though the API persisted the decision, the database did not actually enable replicas, therefore no replica was serving queries.
We released a mitigation last Friday for this to stop happening in new cases, and we rolled out another fix today to retroactively enable replicas for those that have the read replication setting enabled.
If you try out read replication and even after sending some traffic for a few minutes you still don't see replicas serving requests please reach out. You can see if replicas are serving queries in the dashboard UI, and also in each query response from the property
We released a mitigation last Friday for this to stop happening in new cases, and we rolled out another fix today to retroactively enable replicas for those that have the read replication setting enabled.
If you try out read replication and even after sending some traffic for a few minutes you still don't see replicas serving requests please reach out. You can see if replicas are serving queries in the dashboard UI, and also in each query response from the property
served_by_primary: falseHello 
I'm new to this product, and as per the Docs it says data IS encrypted at rest/transit, so does it mean that I could store passwords/sensitive info without any further encryption? Without having for example a Worker to encrypt/decrypst a user password?
I'm new to this product, and as per the Docs it says data IS encrypted at rest/transit, so does it mean that I could store passwords/sensitive info without any further encryption? Without having for example a Worker to encrypt/decrypst a user password?
PYou should still encrypt it
PIt’s encrypted at rest, but there is still an attack vector of if someone gains access to your cloudflare account through a mistake. You should never be able to access someone’s plaintext password period
PTo be more specific, cloudflare encrypts D1 at rest, but passwords should be one way hashed for security which is different
RYup, makes sense
RThanks @PhoenixSpace
PNo problem :)
MSQLite 3.51.0 introduces
jsonb_eachjsonb_treesqlite_version
LIt's SQLite 3.47 right now: https://github.com/cloudflare/workerd/blob/964823ff45df4b5033bae9bba1b43baf6a8c556c/build/deps/deps.MODULE.bazel#L81
We do upgrade SQLite versions but not in a specific cadence. I will ask the team to see if we can upgrade soon again.
We do upgrade SQLite versions but not in a specific cadence. I will ask the team to see if we can upgrade soon again.
LAs long as it doesn't cause writes to the underlying storage of the database, it should retry them, if the error is marked as retryable and the DB is not overloaded. If I remember correctly, temp tables are in memory, so it should be retried.