i was looking at this, although planetscale is pricy i see they do have single nodes now, how effect
i was looking at this, although planetscale is pricy i see they do have single nodes now, how effective is hyperdrive though , is the latency good
LI've heard people on here say that latency was surprisingly good. One person said it was faster than D1 but I think that's because it does some caching. Cloudflare uses Postgres themselves although I expect that they host it themselves.
Mi also heard they use it internally and theres even docs for planetscale, im not sure what to do exactly , it would be nice to stick to cloudflare infra for all things, the thing that im not even sure if user will ever get to 10gb of data, its a second brain type of applicaton, like a advance todo type of appilcation , what i though is that having a db or do for each user would be the best setup but im not really experienced with this so im jsut trying to put together different infras and see how it is.
LDefinitely use DOs not D1 if it's per user. You will struggle with binding otherwise. The nice thing about going the DO route also is that you can encapsulate everything that your client code needs into methods in the DO. For now, they can use the SQLite 10GB DB inside of the DO. You can monitor and see how fast they are growing. If one of your users reaches 5GB in a year, then you'll have a year to migrate the storage part somewhere else, or figure out how to shard. I think you'll be surprised how far 10GB goes. One bad thing about 5GG+ dbs in DOs though is that cold starts can be 1+ seconds if Cloudflare has to move the db file to the machine where the DO runs.
Pyo anyone why dis this happen and how do i fix? pls fm
Pdn
Pdm
PLooking for any gotchas or hidden costs or suprises for D1 global table replication. The website docs seem to say it's 100% free (no additional costs), for example: "Pricing
D1 read replication is built into D1, so you don’t pay extra storage or compute costs for read replicas. You incur the exact same D1 usage billing with or without replicas, based on rows_read and rows_written by your queries." - but how could that be true? If the data is replicated to say 6 other global regions, why would the price be exactly the same as the price for a single instance in a single region? What am I missing?
D1 read replication is built into D1, so you don’t pay extra storage or compute costs for read replicas. You incur the exact same D1 usage billing with or without replicas, based on rows_read and rows_written by your queries." - but how could that be true? If the data is replicated to say 6 other global regions, why would the price be exactly the same as the price for a single instance in a single region? What am I missing?
CThere's a lot of cool information here: https://blog.cloudflare.com/d1-read-replication-beta/
The catch I would say is that you have to send requests (one every 1-2 mins or so) to keep the replica active. Otherwise, it sleeps. If a request is sent from a region with a sleeping replica, it just goes to the primary until it wakes up again. Replicas boot from cold storage and replay writes/etc to get up to date, and then say in sync actively until they go to sleep.
It's not like a constant active-active setup though unless all regions are constantly awake, and the data isn't really replicated until it's active.
I would also take that description as they planned for Read Replication since the start so already baked that into the costs.
The catch I would say is that you have to send requests (one every 1-2 mins or so) to keep the replica active. Otherwise, it sleeps. If a request is sent from a region with a sleeping replica, it just goes to the primary until it wakes up again. Replicas boot from cold storage and replay writes/etc to get up to date, and then say in sync actively until they go to sleep.
It's not like a constant active-active setup though unless all regions are constantly awake, and the data isn't really replicated until it's active.
I would also take that description as they planned for Read Replication since the start so already baked that into the costs.
LWhat Chaika said. D1's pricing included certain number of replicas to be included, hence why there isn't an extra charge. In the future, when there is going to be an arbitrary number of replicas maybe an extra charge will be added for "extras above a certain limit" but this is mostly speculation, we haven't decided anything yet.
Also, you anyway pay for row operations done on replicas, so it's not that you don't give something for using lots of them.
Also, you anyway pay for row operations done on replicas, so it's not that you don't give something for using lots of them.
LAh, also if by "gotchas" you meant anything other than pricing as well, then I would say that the main limitation right now is that replicas are fixed, one per region. This means that unless your load is from all over the world, you will mostly get a a single replica per region helping with the reads, which is not great if most of your traffic is in a single region.
This is something we are planning on extending, hopefully next quarter(s), and allow plenty more replicas per region.
And what Chaika mentioned that without constant traffic replicas hibernate.
This is something we are planning on extending, hopefully next quarter(s), and allow plenty more replicas per region.
And what Chaika mentioned that without constant traffic replicas hibernate.
PJust to clarify, when you say "you anyway pay for row operations done on replicas" - does that mean you pay for all the inserts/updates/deletes that are being syncronized to the read-only copies? Like do you pay for CRUD operations happening in each replicated database instance? OR do you still JUST pay the EXACT SAME price you would for rows written to the primary, NO change if you have replication enabled or not.
POk great to understand the cold start behavior, thank you.
IExact same. 1 read anywhere = 1 read, 1 write on primary = 1 write.
Dhttps://github.com/cloudflare/workers-sdk/issues/5438
It says that last month @penalosa added the feature issue type to this. Does that mean its actively being worked on? Because I basically can't use normal relational database design principles with the sqlite limitations of no major alterations without recreating tables combined with the limitations that defering foreign keys does not defer the ON DELETE CASCADE action
It says that last month @penalosa added the feature issue type to this. Does that mean its actively being worked on? Because I basically can't use normal relational database design principles with the sqlite limitations of no major alterations without recreating tables combined with the limitations that defering foreign keys does not defer the ON DELETE CASCADE action
LYou don't pay for the replicated writes, you pay for all row READS on replicas and primary, and all row WRITES on the primary.
Ma follow on on the durable objects per user setup , i have somewhat implemented it , how do i even test it though ? im not fully versed in dbs in general. my setup is simply a user signups for an account and this automatically sets up a DO for them. thats what i have so far. it works with a single user(me) but i have zero idea how to test it at scale, not that i will have a so many users of the bat, but its good to test, any suggestions
MSomething weird is going with D1 performance for me. Observability shows that every 10 minutes or so, the P99 worker wall time jumps to 30-40 seconds, from a baseline of 100-200ms, and the
duration of d1_batch traces shows the same pattern. Going back the last seven days shows the problem gradually worsening, then fixing itself, then getting worse. My query latency has remained low throughout, at <3ms. My database ID is 585d290a-44f4-4f54-b265-00b35317ae41.
MThe second and third images graph the P99 and P95 durationMS of d1_batch calls.
MNow seeing several
D1 DB storage operation exceeded timeout which caused object to be reset.
THmm I'm a bit confused, or wondering
I have a d1 db with 11MB of data
My requests are very slow but actual sql time is < 1ms, total request time for worker is in the 100s of ms. is that really normal?
The db is in APAC, my requests are coming from APAC and the worker is running in same colo as the db.
I have a d1 db with 11MB of data
My requests are very slow but actual sql time is < 1ms, total request time for worker is in the 100s of ms. is that really normal?
The db is in APAC, my requests are coming from APAC and the worker is running in same colo as the db.
TThe CPU time for the worker is 83ms while walltime is 4.5s
TOf course I will optimise some calls to become parallel calls, but still 200-600ms per call is way higher then I expected
CAPAC is a wide region, you could be sending requests from Indonesia to a D1 DB placed in Tokyo. If you don't see high query latency in your d1 overview, then it's more then likely the latency to it (plus some cold starts if you weren't running a ton of queries)
the worker is running in same colo as the db.Based off what?
TBoth the worker has colo: "SIN" and the D1 instance has colo: "SIN" when checking the raw json in the trace. So my understanding of that would be that they are both running in singapore, and as singapore is a small place (I get 5ms latency pinging 1.1.1.1) I would think that a call within / or across datacenters in singapore should be < 10ms
TAlso I enabled smart placement which I believe should put the worker next to the D1 instance (could be wrong on this)
TAnd for cold start, shouldn't that really just hit the first query in the waterfall
CThat's the colo the request/span is running in, not the location of the D1 db. You can see how it's outside of the d1/binding block, def a bit confusing though
CIn the original version of Smart Placement it did just place it in the same region/location as the D1 DB. Now it's more dynamic/actually looking at latency but should eventually come close-ish if you have enough requests. In your Worker overview you can see Placement Performance, to see if it placed it
and yea cold start would only be after a min or so delay after not sending requests
and yea cold start would only be after a min or so delay after not sending requests
TOk so I enabled read replication and that helped a lot, so the D1 must have been placed in far away land somewhere
TThanks for pointing me in right direction
CIt creates a read-only replica in the same region as the primary, so it was likely just placed closer I'd guess. Interesting though, yea Asia Pacific is a bit of a messy region.
Afaik D1 doesn't actually expose the underlying placement of the instances anywhere, you can only tell by poking each location and looking at latency
Afaik D1 doesn't actually expose the underlying placement of the instances anywhere, you can only tell by poking each location and looking at latency
LWe are planning to add
So, you will have full transparency on where it is located for every query (and where it's moving when it moves). Replicas will also be reporting their own location if they serve queries themselves.
colo airport code in the D1 query responses as well, similar to served_by_region. It's annoying to not know where your DB is located (I know, we should do this sooner!).So, you will have full transparency on where it is located for every query (and where it's moving when it moves). Replicas will also be reporting their own location if they serve queries themselves.
CThat would be super helpful, espec for Asia Pacific region :MeowHeartCloudflare:
MAround 7am UTC, my D1 database started performing normally - with P99(durationMS) falling to ~400ms - but now (7pm UTC) it's getting worse again, P99(durationMS) is 4 seconds and rising. Nothing about my queries is changing, the SQLite query latency is <3ms. Anyone seen this before, and any ideas what to do? Noisy neighbours seems a possibility. There may be some fragmentation issues (I dropped a couple of large tables), but that feels like it would show up in query latency.
TCorrect me if I’m wrong but currently the D1 instances don’t move right? Once created in zone within a region they stay there?
I think it would be helpful to be able to just say run this worker next to my d1 because for APAC especially you end up with the traces above if run far apart. Then if it turns out traffic is primarily coming from somewhere else then they could move in tandem to keep latency down
I think it would be helpful to be able to just say run this worker next to my d1 because for APAC especially you end up with the traces above if run far apart. Then if it turns out traffic is primarily coming from somewhere else then they could move in tandem to keep latency down
Mout of memory: SQLITE_NOMEM
I'm facing this issue, I'm using D1 to keep track of requests for my api, and now I'm getting more and more requests more than 10m records reaching my 10gb limit.
I'm trying to delete them but getting that error out of memory is there any better way to delete them without issues?
I'm facing this issue, I'm using D1 to keep track of requests for my api, and now I'm getting more and more requests more than 10m records reaching my 10gb limit.
I'm trying to delete them but getting that error out of memory is there any better way to delete them without issues?
FYou can try to delete the rows in multiple batches and not all at once.
FThey should stay in the same region, but they can move between different colos
MI'm doing that already, but it's taking so much time trying to delete 10M records
also do you think Workers analytic engine will be good for my usecase, after tracking every day, I group the data and send to another d1, and schedule a cron to delete the rows in batches but again it takes so much time and doesn't delete all the records for that day.
with the analytics Engine if it's possible can we delete all at once (eg for a day 1-2M records) without the limits?
also do you think Workers analytic engine will be good for my usecase, after tracking every day, I group the data and send to another d1, and schedule a cron to delete the rows in batches but again it takes so much time and doesn't delete all the records for that day.
with the analytics Engine if it's possible can we delete all at once (eg for a day 1-2M records) without the limits?
T@Manuel Ogomigo you could try to create a table per day and then just drop it. The benefit of that is I also think you wouldn't pay for the deletes
MOh this is a good idea 
, I’ll work on that. Thank you so much
, I’ll work on that. Thank you so much
PHI, I'm looking for a good way to distinguish between dev mode and production mode.
I'm currently working on a project in Astro and have connected it to d1, but I don't understand how to use local sqlite.
I currently have a drizzle.config.local.ts file with the following :
Is this feasible?
Also because to run in dev I can't use the wrangler cli but
I'm currently working on a project in Astro and have connected it to d1, but I don't understand how to use local sqlite.
I currently have a drizzle.config.local.ts file with the following :
Is this feasible?
Also because to run in dev I can't use the wrangler cli but
astro dev.
Dcan someone explain to me the metrics here? Is it, that this single query caused 12.8k row reads for every execution, or that it caused 12.8k row reads total over 36k executions? If its the first case, why does it cause so many reads, when it should use the primary key here?
FHey, I'm not sure 100% sure, but I think it's the sum for all your queries based on the graphql documentation that is used to retrieve the number of rows read.
Having said that, the best way for you to check would be to
Having said that, the best way for you to check would be to
explain this specific query to find out what sqlite is doing under the hood https://sqlite.org/lang_explain.html.
JHere's how I do it - since the drizzle config is just a typescript file, I run some synchronous code to check for the existence of a
.dev.vars.env.dev.varsdrizzle.config.tsI do the same thing for my Vite-based SPA, in the vite config file I read the
.dev.vars.env
GThat’s what I do as well with durable objects that do 500k+ rows each day, drop the table for those days on a Regular basis and keep the recent ones
GYou can even have a worker cron handle that
CHello Cloudflare developers, I am facing issue with d1 export api. I've also posted details in https://community.cloudflare.com/t/d1-export-api-not-working/862800 Can some one please take a look at this?
MYeah really good idea. U've done that arleady and it's working fine! 