ADoes this let me pin queries? Cloudflare D1 also has a GUI but it's limited. I have some long admin queries I wanna keep around the db instead of my notepad
GYou need to optimize your query, whatever you are doing is more then 128mb which is how much in memory state the d1 durable object has.
Have you tried splitting the reads into paginated requests rather then just selecting all 26 million rows?
Have you tried splitting the reads into paginated requests rather then just selecting all 26 million rows?
1You definitely wanna add the index before having 26mil rows. I would also consider if there are ways to split the data up into separate tables, or only select a subset at a time
LAs others suggested, this is one of the most common ways folks hit the SQLite memory issues with D1 (and Durable Objects). Doing aggregations across the whole table of millions of rows is too easy to hit the memory/CPU limits.
In your specific query:
Although, that would need an index to work properly if you don't already have one, but it's worth trying.
In your specific query:
SELECT calc_date, indicator_id FROM GridData GROUP BY calc_date, indicator_id can you do multiple queries with "data filtering" in ranges so that the rows are filtered out?Although, that would need an index to work properly if you don't already have one, but it's worth trying.
KIsn't hyperdrive and option for you?
LHey folks, a quick update for those that enabled read replication in the past few weeks but didn't see replicas appearing. There was a control plane bug that in some edge cases even though the API persisted the decision, the database did not actually enable replicas, therefore no replica was serving queries.
We released a mitigation last Friday for this to stop happening in new cases, and we rolled out another fix today to retroactively enable replicas for those that have the read replication setting enabled.
If you try out read replication and even after sending some traffic for a few minutes you still don't see replicas serving requests please reach out. You can see if replicas are serving queries in the dashboard UI, and also in each query response from the property
We released a mitigation last Friday for this to stop happening in new cases, and we rolled out another fix today to retroactively enable replicas for those that have the read replication setting enabled.
If you try out read replication and even after sending some traffic for a few minutes you still don't see replicas serving requests please reach out. You can see if replicas are serving queries in the dashboard UI, and also in each query response from the property
served_by_primary: false.
RHello 
I'm new to this product, and as per the Docs it says data IS encrypted at rest/transit, so does it mean that I could store passwords/sensitive info without any further encryption? Without having for example a Worker to encrypt/decrypst a user password?
I'm new to this product, and as per the Docs it says data IS encrypted at rest/transit, so does it mean that I could store passwords/sensitive info without any further encryption? Without having for example a Worker to encrypt/decrypst a user password?
PYou should still encrypt it
PItās encrypted at rest, but there is still an attack vector of if someone gains access to your cloudflare account through a mistake. You should never be able to access someoneās plaintext password period
PTo be more specific, cloudflare encrypts D1 at rest, but passwords should be one way hashed for security which is different
RYup, makes sense
RThanks @PhoenixSpace
PNo problem :)
MSQLite 3.51.0 introduces
jsonb_each and jsonb_tree functions. Does D1 plan to support these? What version of SQLite is D1? sqlite_version is not supported.
LIt's SQLite 3.47 right now: https://github.com/cloudflare/workerd/blob/964823ff45df4b5033bae9bba1b43baf6a8c556c/build/deps/deps.MODULE.bazel#L81
We do upgrade SQLite versions but not in a specific cadence. I will ask the team to see if we can upgrade soon again.
We do upgrade SQLite versions but not in a specific cadence. I will ask the team to see if we can upgrade soon again.
LAs long as it doesn't cause writes to the underlying storage of the database, it should retry them, if the error is marked as retryable and the DB is not overloaded. If I remember correctly, temp tables are in memory, so it should be retried.
C
CAm I doing something wildly wrong?
CHow is cloudflare reporting that there's two entries with the same primary key? Shouldn't that be unique?
CAdditionally, when I try to delete those rows via the studio, it just doesn't, same with editing the values
CC
Ci've gone so far as removing the binding from the worker to ensure that it's not some leftover worker that hasn't terminated or whatever, it's just not saving.
CI've recreated the table, and it's still not letting me save the value via studio
CAh, well, Is ee
CThank you so much haha
CJust swapped it over, working great now, thanks again.
MHow are people handling database backups?
wrangler export was fine when my db was 30Mb (3-4 seconds downtime is acceptable) but now the DB is 500Mb an export takes it offline for ~2 minutes.SOnce a D1 read replica has been created in a region, does it remain there "forever", or is it evicted after a certain period of inactivity. If evicted, what's the policy?
MI believe replicas are based on Durable Objects https://developers.cloudflare.com/durable-objects/concepts/durable-object-lifecycle/#:~:text=While%20in%20the%20idle%2C%20in,transition%20to%20the%20inactive%20state
Cloudflare Docs
This section describes the lifecycle of a Durable Object.
SThat's a good point (I believe all of D1 is based on Durable Objects, not just read replicas). In that case, I guess my question was a bit off... if D1 read replicas follow the DO lifecycle, then they'd stay in memory for a while (as per linked docs). I was hoping to learn whether or not the underlying read replica instance would be cleaned up after a period of inactivity.
LI was hoping to learn whether or not the underlying read replica instance would be cleaned up after a period of inactivity.As the lifecycle doc explains, if there are no requests handled within a region then after some period the replicas will hibernate or get evicted, and the next requests in that region will have to boot the replica again. Those first requests will be forwarded to the primary until the replica is ready again to serve traffic.
In the future we want to allow dynamic replicas that follow your traffic, but the lifecycle will still exist.
SSorry, I think I'm still being unclear. Would the Read Replica's Durable Object be removed from the data centre? My reason for asking was that I'd imagine larger databases would take more time to move and boot and I wondered whether Cloudflare optimised for quicker boots (keep the replica in the data centre) or lower storage (remove the replica from the data centre).
LStill not clear what "removed from the data centre" means.
If you are asking if the replica moves data centers after creation, then at the moment it doesn't unless that specific DC has network issues. But, it will definitely move servers within that data centre. The normal DO routing applies here that attempts to keep a replica on the same DC and server, but it can move anytime due to many reasons under the hood.
Also, replicas need to re-fetch data from cold storage when they are cold starting since they need the latest state in case there were any writes.
If you are asking if the replica moves data centers after creation, then at the moment it doesn't unless that specific DC has network issues. But, it will definitely move servers within that data centre. The normal DO routing applies here that attempts to keep a replica on the same DC and server, but it can move anytime due to many reasons under the hood.
Also, replicas need to re-fetch data from cold storage when they are cold starting since they need the latest state in case there were any writes.
LOverall, your questions seem to suggest that your worry is with low traffic how fast will cold starts be. Then, at the moment, you need to assume actual cold starts all the time. In the future there will be optimizations, but at the moment cold starts are not optimized yet.
If your worry is not about low traffic use-cases, then please provide more details.
If your worry is not about low traffic use-cases, then please provide more details.
SYou're right, my initial worry was about cold starts. I guess there are two scenarios I'm wondering about: when a read replica is first created in a new data centre, and when an existing read replica has to be cold started. Which leaves me with three questions: 1. what's the heuristic for creating a read replica in a new data centre? 2. how quickly can a new read replica be created and booted (not looking for precise numbers, just a way to think about the impact of sqlite storage size)? 3. what affects the cold start time for a read replica (sqlite size, amount of data to be updated, something else)? Apologies if I'm asking silly questions, just trying to get my head around how to think about this.
L- The first a replica (or DO in general) is created in a region it's assigned a specific datacenter location in the region randomly. Then, every time that specific replica comes online it will start there, if that DC is available and healthy, or moved elsewhere in the same region temporarily. This behavior is what happens NOW, and is NOT a guarantee we provide. We want to be flexible to move replicas in the future, since that's the plan, to dynamically create and move replicas based on incoming traffic.
- The main delay is fetching the DB from cold storage. That strictly depends on the size. It can be a few milliseconds or many seconds.
- DB size is the main contributor.
SThis is really useful, thanks. Do read replicas only get created when there's a certain amount of traffic in a given region, and if so, what level of traffic is required? Does the delay scale linearly on DB size and do you have any very rough ballpark delays for say 1GB?
LThe replicas will start booting on the first request to its region, and until they are ready requests are forwarded to the primary. So, if you are only doing 1 request per minute for example, it's unlikely you will get processed by a replica ever, but you will always keep waking it up.
The delay depends on the size yes, I don't have numbers to publicly state, but it's as like downloading N MBs from cold storage.
You can find more details about how it all works under the hood in our blog post: https://blog.cloudflare.com/d1-read-replication-beta/#under-the-hood-how-d1-read-replication-is-implemented
The delay depends on the size yes, I don't have numbers to publicly state, but it's as like downloading N MBs from cold storage.
You can find more details about how it all works under the hood in our blog post: https://blog.cloudflare.com/d1-read-replication-beta/#under-the-hood-how-d1-read-replication-is-implemented
The Cloudflare Blog
D1, Cloudflareās managed SQL database, announces read replication beta. In this post, we do a technical deep dive of the read replication implementation and how your queries can remain consistent across all regions.
Sthanks for your help, much appreciated
TThanks for the great product. Iām hitting an issue with D1 where very simple inserts are taking anywhere from one to eight seconds. Iām only inserting a single user row and returning it. The table has the primary key index and a unique index on email, nothing else. The SQL itself runs in a few milliseconds, but the D1 trace shows long delays before the statement actually completes, which makes it look like platform-level latency rather than anything in the query.
Iām also seeing the same kind of latency spikes on KV puts from time to time. Iām trying to understand whether there are plans to reduce these stalls, or if thereās anything specific I can do on my end to improve D1 insert or KV write performance.
Thanks in advance
Iām also seeing the same kind of latency spikes on KV puts from time to time. Iām trying to understand whether there are plans to reduce these stalls, or if thereās anything specific I can do on my end to improve D1 insert or KV write performance.
Thanks in advance
FHey all I'm excited about the new release of tanstack db (v1 expected to be realsed next month), I'm curious if any one yet used with d1?
Fthere is a concern that progressive mode would get expensive with traditional fetch but I'm thinking that cache-tags would solve this, what do you guys think?
https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-tags/
https://developers.cloudflare.com/cache/how-to/purge-cache/purge-by-tags/
Cloudflare Docs
Cache-tag purging makes multi-file purging easier because you can instantly bulk purge by adding cache-tags to your assets, such as webpages, image files, and more.
FI sent you a DM to get more info, so that I can take a look a your specific D1 DB.
MIs there any indication of when Time Travel might gain the ability to clone and fork databases? Understand you won't have a precise date, but helpful to know if it's closer to "next quarter" or "maybe one day" 
https://developers.cloudflare.com/d1/reference/time-travel/#restore-a-database
At the moment, backups feel like a big missing piece.
https://developers.cloudflare.com/d1/reference/time-travel/#restore-a-database
In the future, D1 will support branching & cloning databases using Time Travel.
At the moment, backups feel like a big missing piece.
- Time travel destructively overwrites the entire database - so restoring one deleted row means losing all good data newer than the bookmark.
wrangler exporttakes the db offline, potentially for multiple minutes depending on size.- I could create a workflow to query tables one by one and output a
.sqlfile, but there could be inconsistent data between tables, foreign key problems, etc.
Maybe I'm missing an option?