@qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D1. i'm both a deskt

@qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D1. i'm both a desktop app and web/saas. in the case of desktop/local users, then they upgrade to paid accounts or if web users want to dowload all of their data or data compliance requires their data be physically located in one geo region, i initially want to keep all user data in a single sqlite db whether local/desktop or web....

DeepOP•9/23/24, 8:13 PM

but since that's not really viable option with current per DB size limit (glad to hear its increasing), i had to re-architect this whole "1 db : 1 user" strategy. but, for me, that is the goal and i'll just stay on top of D1 and see what upgrades happen over time

Acе•9/23/24, 8:14 PM

Yeah I feel that’s one of the main things we’re all waiting for. Cloudflare suggest 1 db per user but give us no easy way to actually do that

DeepOP•9/23/24, 8:15 PM

i see, so i'm not alone

DeepOP•9/23/24, 8:16 PM

yeah, i'm going with turso for now for sqlite bc it's easier to move DBs to edge/geo for customers that need it in EU for instance. and, same kinda deal with them though, the DB size limits / total space GB per account limits make my goal of giving up to 1GB per user in a sqlite DB a non-starter....

DeepOP•9/23/24, 8:18 PM

i may actually just do it myself since i already have edge servers i'm setting up with GPUs around all of the US and then into EU... wouldn't be that hard to then throw a storage server or two into that configuration and just store my own SQLite db's..... but for now, i'm rolling with Turso + CF (KV + R2 for binary/files). kinda a pain in reality but i'll make it work.

[

[FDG] H4kuna•9/24/24, 2:07 PM

Hello, I don't know if anyone can help me or if this is the right channel, but I have a problem with my Linux server (Ubuntu 22.04) when trying to route MariaDB. Could someone help me?

[[FDG] H4kuna Hello, I don't know if anyone can help me or if this is the right channel, but I...

Sam•9/24/24, 2:28 PM

This is not really the server to ask these kind of questions. Maybe in #off-topic ?

Acе•9/24/24, 8:04 PM

kian•9/24/24, 8:16 PM

Different things - SQLite in DOs is the "raw" primitive that D1 (the managed solution) is built on top of.

kian•9/24/24, 8:17 PM

tl;dr D1 is staying at 10 GB (it isn't going down), SQLite in DOs will launch with 1 GB and eventually match at 10 GB

o7•9/24/24, 8:23 PM

is SQLite for DO essentially just synchronous D1 with its input gates

Noitidart•9/25/24, 12:31 PM

Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-database/

And I see I can make reads go to nearest region with session. However the reason I'm looking to move from postgres is because I need writes to go to nearest region. Is it possible for me to associate a region hint with a user ID? Because 99% of time, no one other then that user will ever need to see that writes. Only time is when customer support from different parts of world and other people around world being curious to visit their profile page, in which case it's ok to give them slow response when they hitting the region closest to the user ID of that profile page.

The Cloudflare Blog

Building D1: a Global Database

D1, Cloudflare’s SQL database, is now generally available. With new support for 10GB databases, data export, and enhanced query debugging, we empower developers to build production-ready applications with D1 to meet all their relational SQL needs

NNoitidart Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-da...

boop•9/25/24, 12:33 PM

Read replicas aren't a thing yet. Right now when you create the database, whatever region you specify (or if you didn't specify a region, it will be made in the nearest region to the create request) is where the writes will be sent to, as well as where all the reads will come from

Noitidart•9/25/24, 12:35 PM

Darn. Thanks very much for fast reply. I ran into issues because I'm in San Fran but over time I got a lot of users in UK, South Africa, and Australia. And they all are hitting my SF database leading to major latency.

NNoitidart Darn. Thanks very much for fast reply. I ran into issues because I'm in San Fran...

boop•9/25/24, 12:38 PM

Once read replicas are available, then this should sort out performance issues with reads (except us South Africans who don't have a low latency DO location

). Writes will still go back to the main database location though

Noitidart•9/25/24, 12:41 PM

Haha.

Is there a solution d1 has for writes in the future? Cuz where the latency felt is in the admin panel which is where users spend most of their time. Outside of admin panel I serve state cached using nextjs dyanimc revalidation.

NNoitidart Haha. Is there a solution d1 has for writes in the future? Cuz where the latenc...

boop•9/25/24, 12:44 PM

Since D1 is built on durable objects, until they implement a way for durable objects to move locations dynamically this wont happen. When this feature does become available, the database will be dynamically moved based on where the most of your traffic originates from.

Alllllllllllllll the way at the bottom of this page is where you can find that this feature is planned for Durable Objects in the future, and then D1 should follow not long after that:
https://developers.cloudflare.com/durable-objects/reference/data-location

Noitidart•9/25/24, 12:45 PM

Thanks very much this was very helpful!

Noitidart•9/25/24, 12:49 PM

Relocation though isn't what I would want. As I have users everywhere. Just primary should be considered different depending on user ID. Is thus a possibility with f1 in future?

NNoitidart Relocation though isn't what I would want. As I have users everywhere. Just prim...

lambrospetrou•9/25/24, 1:28 PM

That's the dynamic bindings feature several folks mentioned already above, where you can have different DBs for each of your users. D1 didn't announce anything about it yet.

NNoitidart Relocation though isn't what I would want. As I have users everywhere. Just prim...

boop•9/25/24, 2:21 PM

As @lambrospetrou said, when you create the DB for your user you can set the primary location of that database. Once it's set, you cannot change it

Dynamic bindings is how you'll interact with the database, and you wont be able to supply any location hint at query time, since the database has already been created with your specified parameters. Your write query will travel back to the primary, and your read queries will come from any replicas (once this feature is available) if the data is hot

boop•9/25/24, 2:22 PM

If it operates similarly to how DOs operate, then you won't even have to think about supplying location hints, since the database will be created in the closest region to your users request anyway

Noitidart•9/25/24, 5:24 PM

Thanks guys for this brainstorm. Multiple independent DB's isn't something I can do. I needed one DB, with read and write replicas. Where the write happens also to the nearest DB but would eventually propagate to all others.

TomK•9/26/24, 6:08 AM

I apologize for the interruption, but I am facing a problem and hope to find some solutions. The issue is "D1_ERROR: Network connection lost". I have searched the internet and discord and found that many people have reported this issue but have not found any effective solutions. This problem does not always occur, but sporadically. In the past 24 hours, my worker has only handled 4.5K requests, which is relatively low. So I want to understand whether this is because of my incorrect usage, or due to the instability of the D1 system itself?

Robin Genz•9/26/24, 7:21 AM

Hey

Is there an easy way to seed the local d1 database with data using typescript? I know that I can create a sql file which is then executed by wrangler. But I have a custom typescript file that is called by tsx and uses drizzle + fakerjs and I can't manage to connect to the database. Is there a possibility?

oof2win2•9/26/24, 3:23 PM

By the end of 2024, D1 will support automatic read replication for scalability and low-latency access around the world

hell yeah

Ooof2win2 > By the end of 2024, D1 will support automatic read replication for scalability...

Zack•9/26/24, 3:24 PM

Wait, where did you see this

oof2win2•9/26/24, 3:25 PM

https://blog.cloudflare.com/sqlite-in-durable-objects

The Cloudflare Blog

Zero-latency SQLite storage in every Durable Object

Traditional cloud storage is inherently slow because it is accessed over a network and must synchronize many clients. But what if we could instead put your application code deep into the storage layer, such that your code runs where the data is stored? Durable Objects with SQLite do just that.

Zack•9/26/24, 3:25 PM

Oh nice, hadn't read that one yet

oof2win2•9/26/24, 3:29 PM

honestly what would be nice would be libsql addon support

oof2win2•9/26/24, 3:29 PM

like altering columns and renaming tables and stuff

Ben•9/26/24, 4:25 PM

Yea seems like a great fit for this, could also maybe enable embedded replicas and maybe libSQL server.

Ooof2win2 https://blog.cloudflare.com/sqlite-in-durable-objects

marko.333•9/26/24, 4:50 PM

This is stunning.
I assume (hope) that the DO sqlite is functionally equivalent to D1 (including FTS5 and Json modules)?

BBen Yea seems like a great fit for this, could also maybe enable embedded replicas a...

oof2win2•9/26/24, 5:07 PM

that's probably out of scope and libsql server would mean that there would need to be a connection url - which is not the way cloudflare has it sorted out now

oof2win2•9/26/24, 5:07 PM

just adds overhead, instead an easier way is to make a separate worker that connects via the libsql c/rust client internally

Ooof2win2 that's probably out of scope and libsql server would mean that there would need ...

Ben•9/26/24, 6:08 PM

sorry I thought I was in the durable object channel xD - there are now two sqlite services

Mmarko.333 This is stunning. I assume (hope) that the DO sqlite is functionally equivalent ...

Acе•9/27/24, 12:02 AM

It’s the same stuff that D1 is built on apparently, so those should be there. You could even build your own D1 service with this

DDeep @qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D...

stibbs•9/27/24, 6:05 AM

Is it even possible to specify the region for the db? I assumed if you had regulated data sovereignty concerns D1 was simply not an option? The closest control I'm aware of on this is providing a "location hint" but the docs say

Warning: Providing a location hint does not guarantee that D1 runs in your preferred location. Instead, it will run in the nearest possible location (by latency) to your preference.

Warning: Providing a location hint does not guarantee that D1 runs in your preferred location. Instead, it will run in the nearest possible location (by latency) to your preference.

(source)

Jikyu•9/27/24, 12:58 PM

Would it be possible to use drizzle / run drizzle migrations against the new DO sql database, like we can for D1?

Sstibbs Is it even possible to specify the region for the db? I assumed if you had regul...

Acе•9/27/24, 12:58 PM

Location hints aren’t guaranteed to be spawned in that location because of capacity, datacenter downtime or failures, but very often they will be. You can’t rely on that though. However, now that we have SQL in Durable Objects we can specify a jurisdiction which will always be respected because Durable Objects have that feature.

shixzie•9/27/24, 1:19 PM

no d1 announcements :nootlikethis:

bennycondon•9/27/24, 1:59 PM

I was hoping we’d get an update on dynamic bindings for D1 this week

Bbennycondon I was hoping we’d get an update on dynamic bindings for D1 this week

boop•9/27/24, 2:25 PM

Same, it's been over 2 years since D1 was announced and still no way to use it with workers the way it was intended

shixzie•9/27/24, 2:42 PM

yup, i guess turso will be the choice for multi-tenant apps rn

LordSilver•9/27/24, 2:42 PM

Is D1 a closed-source fork of SQLite, something similar to libSQL? Is it 100% compatible with SQLite? If you wanted to export your D1 db and move it to any other SQLite hosting, would you be able to do that?

LLordSilver Is D1 a closed-source fork of SQLite, something similar to libSQL? Is it 100% co...

Hard@Work•9/27/24, 2:43 PM

The runtime component of it should be in https://github.com/cloudflare/workerd

GitHub

GitHub - cloudflare/workerd: The JavaScript / Wasm runtime that pow...

The JavaScript / Wasm runtime that powers Cloudflare Workers - cloudflare/workerd

Hard@Work•9/27/24, 2:43 PM

Somewhere

Sshixzie yup, i guess turso will be the choice for multi-tenant apps rn

kooliebwoykg•9/27/24, 2:44 PM

I switched and couldn't believe how easy it was and how beautiful multi-tenant works with Turso.

Sstibbs Is it even possible to specify the region for the db? I assumed if you had regul...

DeepOP•9/27/24, 3:08 PM

you're right. i've dropped D1 as an option for me for other reasons as well.

AAcе Location hints aren’t guaranteed to be spawned in that location because of capac...

lambrospetrou•9/27/24, 4:24 PM

Just to clarify on this, location hints in D1 work in similar way to location hints in DOs, since D1 uses SQLite-in-DOs under the hood.

Jurisdictions is a different thing, making sure your DO is within a specific broader area, with only

eu

eu

and

fedramp

fedramp

as possible values right now (see docs).

@qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D1. i'm both a deskt

Similar Threads