What would be your use case for restoring a database via an HTTP api call?
What would be your use case for restoring a database via an HTTP api call?
EI've been talking about it with lambrospetrou in #durable-objects .
Quick summary: I'm processing live data and sometimes that data is faulty and I need to revert to an earlier point and replay the corrected data.
With durable objects that's possible at runtime. But I also have D1s.
I'm trying to avoid writing revert logic for all the incoming data since this is difficult to maintain.
Quick summary: I'm processing live data and sometimes that data is faulty and I need to revert to an earlier point and replay the corrected data.
With durable objects that's possible at runtime. But I also have D1s.
I'm trying to avoid writing revert logic for all the incoming data since this is difficult to maintain.
PDoes D1 have triggers for tables? Like can I raise an event to a Queue when a insert/update/delete happens?
EOnly internal SQLite triggers, you can't call outside. Maybe durable objects would be a better fit?
LThis feature is commonly called CDC (change data capture), and it's not supported at the moment. The SQLite triggers feature is supported, but that's only to execute SQL things when certain rows change, so not what you are asking.
HDumb idea, use SQLite triggers to push to a triggers table, and then periodically poll it?
AI am facing an issue where I am unable to access some parts of my site
For example, when I try to remove something from the cart, it throws an error and does not let me remove it, additionally it does not let me logout
When used in development mode, it works fine
I purged cache and all but still not working, any insights?
For example, when I try to remove something from the cart, it throws an error and does not let me remove it, additionally it does not let me logout
When used in development mode, it works fine
I purged cache and all but still not working, any insights?
Mhi all is it possible to use d1 per user, the limit is 50k databases what if i have morethen 50k users , can you increase it and also its 0.75 dollars for a gb after 5gb does this inclulde each database ?
DI'm curious, why a separate DB per user?
Kuse durable objects for per user storage. there is no limit on durable objects count. it uses the same storage as D1.
LThe limit can be increased, yes, but it's going to be a pain to bind them all to the same worker. If you want a single worker to have access to thousands/millions of DBs, as others suggested using SQLite Durable Objects is the recommended approach.
LAre you using D1, and if yes, are you seeing errors from D1? Otherwise you are probably going to get more help from the right channel for what's failing in your setup.
CHi, my D1 queries aren't working with Vitest. Is there a solution for this?
Mappreacite the feedback, i needed much more storage for my db so i had a look around and found hyperdrive and planetscale, would this be a good setup , im using nextjs with cloudflare workers everything works but i need a scalable db.
Mplanetscale is pricy though , how scalable and effective is this setup that you have reccomended , is per user db even a good approach , i know i will need way more then 10gb of storage but i dont really know which approach to use
LIf you need more than 10gb per user, don't go the DO route. Sharding will be complexity you probably don't want to deal with. I recommend using Cloudflare Hyperdrive to connect to a Postgres database hosted somewhere. https://developers.cloudflare.com/workers/tutorials/postgres/
Cloudflare Docs
This tutorial explains how to connect to a Postgres database with Cloudflare Workers. The Workers application you create in this tutorial will interact with a product database inside of Postgres.
Mi was looking at this, although planetscale is pricy i see they do have single nodes now, how effective is hyperdrive though , is the latency good
LI've heard people on here say that latency was surprisingly good. One person said it was faster than D1 but I think that's because it does some caching. Cloudflare uses Postgres themselves although I expect that they host it themselves.
Mi also heard they use it internally and theres even docs for planetscale, im not sure what to do exactly , it would be nice to stick to cloudflare infra for all things, the thing that im not even sure if user will ever get to 10gb of data, its a second brain type of applicaton, like a advance todo type of appilcation , what i though is that having a db or do for each user would be the best setup but im not really experienced with this so im jsut trying to put together different infras and see how it is.
LDefinitely use DOs not D1 if it's per user. You will struggle with binding otherwise. The nice thing about going the DO route also is that you can encapsulate everything that your client code needs into methods in the DO. For now, they can use the SQLite 10GB DB inside of the DO. You can monitor and see how fast they are growing. If one of your users reaches 5GB in a year, then you'll have a year to migrate the storage part somewhere else, or figure out how to shard. I think you'll be surprised how far 10GB goes. One bad thing about 5GG+ dbs in DOs though is that cold starts can be 1+ seconds if Cloudflare has to move the db file to the machine where the DO runs.
Pyo anyone why dis this happen and how do i fix? pls fm
Pdn
Pdm
PLooking for any gotchas or hidden costs or suprises for D1 global table replication. The website docs seem to say it's 100% free (no additional costs), for example: "Pricing
D1 read replication is built into D1, so you don’t pay extra storage or compute costs for read replicas. You incur the exact same D1 usage billing with or without replicas, based on rows_read and rows_written by your queries." - but how could that be true? If the data is replicated to say 6 other global regions, why would the price be exactly the same as the price for a single instance in a single region? What am I missing?
D1 read replication is built into D1, so you don’t pay extra storage or compute costs for read replicas. You incur the exact same D1 usage billing with or without replicas, based on rows_read and rows_written by your queries." - but how could that be true? If the data is replicated to say 6 other global regions, why would the price be exactly the same as the price for a single instance in a single region? What am I missing?
CThere's a lot of cool information here: https://blog.cloudflare.com/d1-read-replication-beta/
The catch I would say is that you have to send requests (one every 1-2 mins or so) to keep the replica active. Otherwise, it sleeps. If a request is sent from a region with a sleeping replica, it just goes to the primary until it wakes up again. Replicas boot from cold storage and replay writes/etc to get up to date, and then say in sync actively until they go to sleep.
It's not like a constant active-active setup though unless all regions are constantly awake, and the data isn't really replicated until it's active.
I would also take that description as they planned for Read Replication since the start so already baked that into the costs.
The catch I would say is that you have to send requests (one every 1-2 mins or so) to keep the replica active. Otherwise, it sleeps. If a request is sent from a region with a sleeping replica, it just goes to the primary until it wakes up again. Replicas boot from cold storage and replay writes/etc to get up to date, and then say in sync actively until they go to sleep.
It's not like a constant active-active setup though unless all regions are constantly awake, and the data isn't really replicated until it's active.
I would also take that description as they planned for Read Replication since the start so already baked that into the costs.
LWhat Chaika said. D1's pricing included certain number of replicas to be included, hence why there isn't an extra charge. In the future, when there is going to be an arbitrary number of replicas maybe an extra charge will be added for "extras above a certain limit" but this is mostly speculation, we haven't decided anything yet.
Also, you anyway pay for row operations done on replicas, so it's not that you don't give something for using lots of them.
Also, you anyway pay for row operations done on replicas, so it's not that you don't give something for using lots of them.
LAh, also if by "gotchas" you meant anything other than pricing as well, then I would say that the main limitation right now is that replicas are fixed, one per region. This means that unless your load is from all over the world, you will mostly get a a single replica per region helping with the reads, which is not great if most of your traffic is in a single region.
This is something we are planning on extending, hopefully next quarter(s), and allow plenty more replicas per region.
And what Chaika mentioned that without constant traffic replicas hibernate.
This is something we are planning on extending, hopefully next quarter(s), and allow plenty more replicas per region.
And what Chaika mentioned that without constant traffic replicas hibernate.
PJust to clarify, when you say "you anyway pay for row operations done on replicas" - does that mean you pay for all the inserts/updates/deletes that are being syncronized to the read-only copies? Like do you pay for CRUD operations happening in each replicated database instance? OR do you still JUST pay the EXACT SAME price you would for rows written to the primary, NO change if you have replication enabled or not.
POk great to understand the cold start behavior, thank you.
IExact same. 1 read anywhere = 1 read, 1 write on primary = 1 write.
Dhttps://github.com/cloudflare/workers-sdk/issues/5438
It says that last month @penalosa added the feature issue type to this. Does that mean its actively being worked on? Because I basically can't use normal relational database design principles with the sqlite limitations of no major alterations without recreating tables combined with the limitations that defering foreign keys does not defer the ON DELETE CASCADE action
It says that last month @penalosa added the feature issue type to this. Does that mean its actively being worked on? Because I basically can't use normal relational database design principles with the sqlite limitations of no major alterations without recreating tables combined with the limitations that defering foreign keys does not defer the ON DELETE CASCADE action
LYou don't pay for the replicated writes, you pay for all row READS on replicas and primary, and all row WRITES on the primary.
Ma follow on on the durable objects per user setup , i have somewhat implemented it , how do i even test it though ? im not fully versed in dbs in general. my setup is simply a user signups for an account and this automatically sets up a DO for them. thats what i have so far. it works with a single user(me) but i have zero idea how to test it at scale, not that i will have a so many users of the bat, but its good to test, any suggestions
MSomething weird is going with D1 performance for me. Observability shows that every 10 minutes or so, the P99 worker wall time jumps to 30-40 seconds, from a baseline of 100-200ms, and the
duration of d1_batch traces shows the same pattern. Going back the last seven days shows the problem gradually worsening, then fixing itself, then getting worse. My query latency has remained low throughout, at <3ms. My database ID is 585d290a-44f4-4f54-b265-00b35317ae41.
MThe second and third images graph the P99 and P95 durationMS of d1_batch calls.
MNow seeing several
D1 DB storage operation exceeded timeout which caused object to be reset.
THmm I'm a bit confused, or wondering
I have a d1 db with 11MB of data
My requests are very slow but actual sql time is < 1ms, total request time for worker is in the 100s of ms. is that really normal?
The db is in APAC, my requests are coming from APAC and the worker is running in same colo as the db.
I have a d1 db with 11MB of data
My requests are very slow but actual sql time is < 1ms, total request time for worker is in the 100s of ms. is that really normal?
The db is in APAC, my requests are coming from APAC and the worker is running in same colo as the db.
TThe CPU time for the worker is 83ms while walltime is 4.5s
TOf course I will optimise some calls to become parallel calls, but still 200-600ms per call is way higher then I expected
CAPAC is a wide region, you could be sending requests from Indonesia to a D1 DB placed in Tokyo. If you don't see high query latency in your d1 overview, then it's more then likely the latency to it (plus some cold starts if you weren't running a ton of queries)
the worker is running in same colo as the db.Based off what?
TBoth the worker has colo: "SIN" and the D1 instance has colo: "SIN" when checking the raw json in the trace. So my understanding of that would be that they are both running in singapore, and as singapore is a small place (I get 5ms latency pinging 1.1.1.1) I would think that a call within / or across datacenters in singapore should be < 10ms
TAlso I enabled smart placement which I believe should put the worker next to the D1 instance (could be wrong on this)
TAnd for cold start, shouldn't that really just hit the first query in the waterfall
CThat's the colo the request/span is running in, not the location of the D1 db. You can see how it's outside of the d1/binding block, def a bit confusing though
CIn the original version of Smart Placement it did just place it in the same region/location as the D1 DB. Now it's more dynamic/actually looking at latency but should eventually come close-ish if you have enough requests. In your Worker overview you can see Placement Performance, to see if it placed it
and yea cold start would only be after a min or so delay after not sending requests
and yea cold start would only be after a min or so delay after not sending requests
TOk so I enabled read replication and that helped a lot, so the D1 must have been placed in far away land somewhere
TThanks for pointing me in right direction
CIt creates a read-only replica in the same region as the primary, so it was likely just placed closer I'd guess. Interesting though, yea Asia Pacific is a bit of a messy region.
Afaik D1 doesn't actually expose the underlying placement of the instances anywhere, you can only tell by poking each location and looking at latency
Afaik D1 doesn't actually expose the underlying placement of the instances anywhere, you can only tell by poking each location and looking at latency
LWe are planning to add
So, you will have full transparency on where it is located for every query (and where it's moving when it moves). Replicas will also be reporting their own location if they serve queries themselves.
colo airport code in the D1 query responses as well, similar to served_by_region. It's annoying to not know where your DB is located (I know, we should do this sooner!).So, you will have full transparency on where it is located for every query (and where it's moving when it moves). Replicas will also be reporting their own location if they serve queries themselves.
CThat would be super helpful, espec for Asia Pacific region :MeowHeartCloudflare:
MAround 7am UTC, my D1 database started performing normally - with P99(durationMS) falling to ~400ms - but now (7pm UTC) it's getting worse again, P99(durationMS) is 4 seconds and rising. Nothing about my queries is changing, the SQLite query latency is <3ms. Anyone seen this before, and any ideas what to do? Noisy neighbours seems a possibility. There may be some fragmentation issues (I dropped a couple of large tables), but that feels like it would show up in query latency.
