Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-database/ And I see I

Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-database/

And I see I can make reads go to nearest region with session. However the reason I'm looking to move from postgres is because I need writes to go to nearest region. Is it possible for me to associate a region hint with a user ID? Because 99% of time, no one other then that user will ever need to see that writes. Only time is when customer support from different parts of world and other people around world being curious to visit their profile page, in which case it's ok to give them slow response when they hitting the region closest to the user ID of that profile page.

The Cloudflare Blog

Building D1: a Global Database

D1, Cloudflare’s SQL database, is now generally available. With new support for 10GB databases, data export, and enhanced query debugging, we empower developers to build production-ready applications with D1 to meet all their relational SQL needs

NNoitidart Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-da...

boop•9/25/24, 12:33 PM

Read replicas aren't a thing yet. Right now when you create the database, whatever region you specify (or if you didn't specify a region, it will be made in the nearest region to the create request) is where the writes will be sent to, as well as where all the reads will come from

NoitidartOP•9/25/24, 12:35 PM

Darn. Thanks very much for fast reply. I ran into issues because I'm in San Fran but over time I got a lot of users in UK, South Africa, and Australia. And they all are hitting my SF database leading to major latency.

NNoitidart Darn. Thanks very much for fast reply. I ran into issues because I'm in San Fran...

boop•9/25/24, 12:38 PM

Once read replicas are available, then this should sort out performance issues with reads (except us South Africans who don't have a low latency DO location

). Writes will still go back to the main database location though

NoitidartOP•9/25/24, 12:41 PM

Haha.

Is there a solution d1 has for writes in the future? Cuz where the latency felt is in the admin panel which is where users spend most of their time. Outside of admin panel I serve state cached using nextjs dyanimc revalidation.

NNoitidart Haha. Is there a solution d1 has for writes in the future? Cuz where the latenc...

boop•9/25/24, 12:44 PM

Since D1 is built on durable objects, until they implement a way for durable objects to move locations dynamically this wont happen. When this feature does become available, the database will be dynamically moved based on where the most of your traffic originates from.

Alllllllllllllll the way at the bottom of this page is where you can find that this feature is planned for Durable Objects in the future, and then D1 should follow not long after that:
https://developers.cloudflare.com/durable-objects/reference/data-location

NoitidartOP•9/25/24, 12:45 PM

Thanks very much this was very helpful!

NoitidartOP•9/25/24, 12:49 PM

Relocation though isn't what I would want. As I have users everywhere. Just primary should be considered different depending on user ID. Is thus a possibility with f1 in future?

NNoitidart Relocation though isn't what I would want. As I have users everywhere. Just prim...

lambrospetrou•9/25/24, 1:28 PM

That's the dynamic bindings feature several folks mentioned already above, where you can have different DBs for each of your users. D1 didn't announce anything about it yet.

NNoitidart Relocation though isn't what I would want. As I have users everywhere. Just prim...

boop•9/25/24, 2:21 PM

As @lambrospetrou said, when you create the DB for your user you can set the primary location of that database. Once it's set, you cannot change it

Dynamic bindings is how you'll interact with the database, and you wont be able to supply any location hint at query time, since the database has already been created with your specified parameters. Your write query will travel back to the primary, and your read queries will come from any replicas (once this feature is available) if the data is hot

boop•9/25/24, 2:22 PM

If it operates similarly to how DOs operate, then you won't even have to think about supplying location hints, since the database will be created in the closest region to your users request anyway

NoitidartOP•9/25/24, 5:24 PM

Thanks guys for this brainstorm. Multiple independent DB's isn't something I can do. I needed one DB, with read and write replicas. Where the write happens also to the nearest DB but would eventually propagate to all others.

TomK•9/26/24, 6:08 AM

I apologize for the interruption, but I am facing a problem and hope to find some solutions. The issue is "D1_ERROR: Network connection lost". I have searched the internet and discord and found that many people have reported this issue but have not found any effective solutions. This problem does not always occur, but sporadically. In the past 24 hours, my worker has only handled 4.5K requests, which is relatively low. So I want to understand whether this is because of my incorrect usage, or due to the instability of the D1 system itself?

Robin Genz•9/26/24, 7:21 AM

Hey

Is there an easy way to seed the local d1 database with data using typescript? I know that I can create a sql file which is then executed by wrangler. But I have a custom typescript file that is called by tsx and uses drizzle + fakerjs and I can't manage to connect to the database. Is there a possibility?

oof2win2•9/26/24, 3:23 PM

By the end of 2024, D1 will support automatic read replication for scalability and low-latency access around the world

hell yeah

Ooof2win2 > By the end of 2024, D1 will support automatic read replication for scalability...

Zack•9/26/24, 3:24 PM

Wait, where did you see this

oof2win2•9/26/24, 3:25 PM

https://blog.cloudflare.com/sqlite-in-durable-objects

The Cloudflare Blog

Zero-latency SQLite storage in every Durable Object

Traditional cloud storage is inherently slow because it is accessed over a network and must synchronize many clients. But what if we could instead put your application code deep into the storage layer, such that your code runs where the data is stored? Durable Objects with SQLite do just that.

Zack•9/26/24, 3:25 PM

Oh nice, hadn't read that one yet

oof2win2•9/26/24, 3:29 PM

honestly what would be nice would be libsql addon support

oof2win2•9/26/24, 3:29 PM

like altering columns and renaming tables and stuff

Ben•9/26/24, 4:25 PM

Yea seems like a great fit for this, could also maybe enable embedded replicas and maybe libSQL server.

Ooof2win2 https://blog.cloudflare.com/sqlite-in-durable-objects

marko.333•9/26/24, 4:50 PM

This is stunning.
I assume (hope) that the DO sqlite is functionally equivalent to D1 (including FTS5 and Json modules)?

BBen Yea seems like a great fit for this, could also maybe enable embedded replicas a...

oof2win2•9/26/24, 5:07 PM

that's probably out of scope and libsql server would mean that there would need to be a connection url - which is not the way cloudflare has it sorted out now

oof2win2•9/26/24, 5:07 PM

just adds overhead, instead an easier way is to make a separate worker that connects via the libsql c/rust client internally

Ooof2win2 that's probably out of scope and libsql server would mean that there would need ...

Ben•9/26/24, 6:08 PM

sorry I thought I was in the durable object channel xD - there are now two sqlite services

Mmarko.333 This is stunning. I assume (hope) that the DO sqlite is functionally equivalent ...

Acе•9/27/24, 12:02 AM

It’s the same stuff that D1 is built on apparently, so those should be there. You could even build your own D1 service with this

DDeep @qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D...

stibbs•9/27/24, 6:05 AM

Is it even possible to specify the region for the db? I assumed if you had regulated data sovereignty concerns D1 was simply not an option? The closest control I'm aware of on this is providing a "location hint" but the docs say

Warning: Providing a location hint does not guarantee that D1 runs in your preferred location. Instead, it will run in the nearest possible location (by latency) to your preference.

Warning: Providing a location hint does not guarantee that D1 runs in your preferred location. Instead, it will run in the nearest possible location (by latency) to your preference.

(source)

Jikyu•9/27/24, 12:58 PM

Would it be possible to use drizzle / run drizzle migrations against the new DO sql database, like we can for D1?

Sstibbs Is it even possible to specify the region for the db? I assumed if you had regul...

Acе•9/27/24, 12:58 PM

Location hints aren’t guaranteed to be spawned in that location because of capacity, datacenter downtime or failures, but very often they will be. You can’t rely on that though. However, now that we have SQL in Durable Objects we can specify a jurisdiction which will always be respected because Durable Objects have that feature.

shixzie•9/27/24, 1:19 PM

no d1 announcements :nootlikethis:

bennycondon•9/27/24, 1:59 PM

I was hoping we’d get an update on dynamic bindings for D1 this week

Bbennycondon I was hoping we’d get an update on dynamic bindings for D1 this week

boop•9/27/24, 2:25 PM

Same, it's been over 2 years since D1 was announced and still no way to use it with workers the way it was intended

shixzie•9/27/24, 2:42 PM

yup, i guess turso will be the choice for multi-tenant apps rn

LordSilver•9/27/24, 2:42 PM

Is D1 a closed-source fork of SQLite, something similar to libSQL? Is it 100% compatible with SQLite? If you wanted to export your D1 db and move it to any other SQLite hosting, would you be able to do that?

LLordSilver Is D1 a closed-source fork of SQLite, something similar to libSQL? Is it 100% co...

Hard@Work•9/27/24, 2:43 PM

The runtime component of it should be in https://github.com/cloudflare/workerd

GitHub

GitHub - cloudflare/workerd: The JavaScript / Wasm runtime that pow...

The JavaScript / Wasm runtime that powers Cloudflare Workers - cloudflare/workerd

Hard@Work•9/27/24, 2:43 PM

Somewhere

Sshixzie yup, i guess turso will be the choice for multi-tenant apps rn

kooliebwoykg•9/27/24, 2:44 PM

I switched and couldn't believe how easy it was and how beautiful multi-tenant works with Turso.

Sstibbs Is it even possible to specify the region for the db? I assumed if you had regul...

Deep•9/27/24, 3:08 PM

you're right. i've dropped D1 as an option for me for other reasons as well.

AAcе Location hints aren’t guaranteed to be spawned in that location because of capac...

lambrospetrou•9/27/24, 4:24 PM

Just to clarify on this, location hints in D1 work in similar way to location hints in DOs, since D1 uses SQLite-in-DOs under the hood.

Jurisdictions is a different thing, making sure your DO is within a specific broader area, with only

eu

eu

and

fedramp

fedramp

as possible values right now (see docs).

Bboop Same, it's been over 2 years since D1 was announced and still no way to use it w...

lambrospetrou•9/27/24, 4:36 PM

Can't share timelines yet, but we are working on some exciting features to improve this that will be coming over the next few months!

Llambrospetrou Can't share timelines yet, but we are working on some exciting features to impro...

boop•9/27/24, 5:55 PM

Is "exciting features" dynamic bindings? I'm bored of waiting for a feature that is a requirement to using D1 properly. Redeploying my worker every time I get a new customer to get past the 10gb limitation using an external service is just plain stupid. The D1 team should probably ask the DO team if they can borrow their binding code

Bboop Is "exciting features" dynamic bindings? I'm bored of waiting for a feature that...

o7•9/27/24, 6:51 PM

Both teams are obviously already working together, D1 is built on SQLite for DO. Also, I don't want to get too schizo, but I am 100% sure that they're releasing it in the next few days given the context clues

o7•9/27/24, 6:55 PM

if im wrong i will delete my account and never be seen here again

Oo7 if im wrong i will delete my account and never be seen here again

shixzie•9/27/24, 7:59 PM

these kind of statements don't help, I myself like what Cloudflare is doing in general and of course would like to have features now, but making a case for why they didn't even show a sneak-peak when they have the resources needed to do whatever it takes to at least keep the hype going is not good.

shixzie•9/27/24, 7:59 PM

Personally, I was waiting for the multi-tenant feature or at least a sneak as to how it would be implemented to plan ahead for some future projects. But I can only rely on what's available today, leaving no other choice tbh.

Somepotato•9/27/24, 8:22 PM

This was last asked awhile ago but is there any news about integrating the spatialite extension within D1?

Oo7 Both teams are obviously already working together, D1 is built on SQLite for DO....

boop•9/27/24, 8:32 PM

Yeah I know, that was more a frustrated dig at the D1 team. The “hey it’s coming soon” is getting boring now and just makes it look like an unfinished product

Sshixzie these kind of statements don't help, I myself like what Cloudflare is doing in g...

o7•9/27/24, 8:32 PM

Original message was deleted

o7•9/27/24, 8:40 PM

The 10GB limit is more of an architectural limit (they probably could go a little higher but 10GB is a safe limit per DB)

Llambrospetrou Just to clarify on this, location hints in D1 work in similar way to location hi...

stibbs•9/28/24, 2:45 AM

Jurisdictions is what I need, but more options -- specifically Australia

�

𝓈𝒽𝓎•9/28/24, 8:23 AM

D1 without multi tenancy features and sharding with 10GB limit, regardless what people say about how high it is and whatever, what happens when I get there? do I just shut down the company once I reach 10GB limit?

Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-database/ And I see I

Similar Threads

Similar Threads

Similar Threads