Hey 👋🏻 Quick question: Is there an easy way to find the most „expensive“ (most reads/writes) queri

Hey

Quick question: Is there an easy way to find the most „expensive“ (most reads/writes) queries in my code (e.g. a list in the Cloudflare dashboard) without manually checking the meta data of every single query?

RRobin Genz Hey 👋🏻 Quick question: Is there an easy way to find the most „expensive“ (most...

boop•9/21/24, 10:10 AM

https://developers.cloudflare.com/d1/observability/metrics-analytics/#query-insights

Cloudflare Docs

Metrics and analytics | Cloudflare D1 docs

D1 exposes database analytics that allow you to inspect query volume, query latency, and storage size across all and/or each database in your account.

naourass•9/23/24, 2:58 AM

Hi all, I'm using Astro.build with the Cloudflare integration and I'm trying to add a staging and a prod database beside the local dev one. Anyone has an idea for how to achieve that so the api endpoints would pick the local db when in dev environment, and the staging or prod db when in the staging or prod db respectively? I've added a different db binding under each environment in

wrangler.toml

wrangler.toml

, but I have no idea and can't find any info for how to make the api endpoint pick the right db on each environment (using

locals.runtime.env.DB

locals.runtime.env.DB

celer•9/23/24, 12:41 PM

Howdy y'all! I have a D1 DB with several tables, which have some foreign keys to each other (never making a cycle). Sometimes I need to recreate the DB (for testing purposes), but I don't want to regenerate db_id (because it's in wrangler.toml), so I want to drop all tables and recreate them. However, querying all table names and generating/executing a script to drop one-by-one always fails on FK related error. Even when there's no data in DB. Even when using PRAGMA defer_foreign_keys=true. Looks like the only way is to construct a dependency graph and drop tables in correct order, so there's never a FK poiting to already dropped table.
Does anyone know if there is a way to work around this and drop all tables at once without FK check? Thanks!

Deep•9/23/24, 3:48 PM

does anyone know if the 50k databases limit is a hard limit per account? or when we hit 50k databases (users) can we then upgrade to enterprise account for unlimited number of databases?

DDeep does anyone know if the 50k databases limit is a hard limit per account? or whe...

o7•9/23/24, 3:50 PM

d1-database also dynamic bindings are coming soon and they probably correlate with a higher number of dbs (natively without having to contact support for a higher limit (but thats just my speculation))

Oo7 [soft limit](https://discord.com/channels/595317990191398933/992060581832032316/...

Acе•9/23/24, 7:37 PM

From what I’ve seen, that’s likely coming out this week, there will be a 1GB database size limit (10GB after beta), and you can make unlimited. It will be that a Durable Object can have a SQL database for storage. You’ll be able to choose whether a DO class uses KV storage or SQL.

AAcе From what I’ve seen, that’s likely coming out this week, there will be a 1GB dat...

Isaac McFadyen•9/23/24, 7:41 PM

From the previous blog posts I believe DO SQLite will be a seperate opt-in thing, not the same as D1 dynamic bindings.

Isaac McFadyen•9/23/24, 7:41 PM

More for people who want low-level access to SQLite.

Acе•9/23/24, 7:50 PM

Ah yeah that makes sense, I’m sure the DO thing will solve most of our use cases in the meantime. I imagine dynamic D1 would work similar to how Workers for Platforms is essentially just dynamic service bindings

Deep•9/23/24, 8:12 PM

@qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D1. i'm both a desktop app and web/saas. in the case of desktop/local users, then they upgrade to paid accounts or if web users want to dowload all of their data or data compliance requires their data be physically located in one geo region, i initially want to keep all user data in a single sqlite db whether local/desktop or web....

Deep•9/23/24, 8:13 PM

but since that's not really viable option with current per DB size limit (glad to hear its increasing), i had to re-architect this whole "1 db : 1 user" strategy. but, for me, that is the goal and i'll just stay on top of D1 and see what upgrades happen over time

Acе•9/23/24, 8:14 PM

Yeah I feel that’s one of the main things we’re all waiting for. Cloudflare suggest 1 db per user but give us no easy way to actually do that

Deep•9/23/24, 8:15 PM

i see, so i'm not alone

Deep•9/23/24, 8:16 PM

yeah, i'm going with turso for now for sqlite bc it's easier to move DBs to edge/geo for customers that need it in EU for instance. and, same kinda deal with them though, the DB size limits / total space GB per account limits make my goal of giving up to 1GB per user in a sqlite DB a non-starter....

Deep•9/23/24, 8:18 PM

i may actually just do it myself since i already have edge servers i'm setting up with GPUs around all of the US and then into EU... wouldn't be that hard to then throw a storage server or two into that configuration and just store my own SQLite db's..... but for now, i'm rolling with Turso + CF (KV + R2 for binary/files). kinda a pain in reality but i'll make it work.

[

[FDG] H4kuna•9/24/24, 2:07 PM

Hello, I don't know if anyone can help me or if this is the right channel, but I have a problem with my Linux server (Ubuntu 22.04) when trying to route MariaDB. Could someone help me?

[[FDG] H4kuna Hello, I don't know if anyone can help me or if this is the right channel, but I...

Sam•9/24/24, 2:28 PM

This is not really the server to ask these kind of questions. Maybe in #off-topic ?

Acе•9/24/24, 8:04 PM

kian•9/24/24, 8:16 PM

Different things - SQLite in DOs is the "raw" primitive that D1 (the managed solution) is built on top of.

kian•9/24/24, 8:17 PM

tl;dr D1 is staying at 10 GB (it isn't going down), SQLite in DOs will launch with 1 GB and eventually match at 10 GB

o7•9/24/24, 8:23 PM

is SQLite for DO essentially just synchronous D1 with its input gates

Noitidart•9/25/24, 12:31 PM

Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-database/

And I see I can make reads go to nearest region with session. However the reason I'm looking to move from postgres is because I need writes to go to nearest region. Is it possible for me to associate a region hint with a user ID? Because 99% of time, no one other then that user will ever need to see that writes. Only time is when customer support from different parts of world and other people around world being curious to visit their profile page, in which case it's ok to give them slow response when they hitting the region closest to the user ID of that profile page.

The Cloudflare Blog

Building D1: a Global Database

D1, Cloudflare’s SQL database, is now generally available. With new support for 10GB databases, data export, and enhanced query debugging, we empower developers to build production-ready applications with D1 to meet all their relational SQL needs

NNoitidart Hi all, I was reading this - https://blog.cloudflare.com/building-d1-a-global-da...

boop•9/25/24, 12:33 PM

Read replicas aren't a thing yet. Right now when you create the database, whatever region you specify (or if you didn't specify a region, it will be made in the nearest region to the create request) is where the writes will be sent to, as well as where all the reads will come from

Noitidart•9/25/24, 12:35 PM

Darn. Thanks very much for fast reply. I ran into issues because I'm in San Fran but over time I got a lot of users in UK, South Africa, and Australia. And they all are hitting my SF database leading to major latency.

NNoitidart Darn. Thanks very much for fast reply. I ran into issues because I'm in San Fran...

boop•9/25/24, 12:38 PM

Once read replicas are available, then this should sort out performance issues with reads (except us South Africans who don't have a low latency DO location

). Writes will still go back to the main database location though

Noitidart•9/25/24, 12:41 PM

Haha.

Is there a solution d1 has for writes in the future? Cuz where the latency felt is in the admin panel which is where users spend most of their time. Outside of admin panel I serve state cached using nextjs dyanimc revalidation.

NNoitidart Haha. Is there a solution d1 has for writes in the future? Cuz where the latenc...

boop•9/25/24, 12:44 PM

Since D1 is built on durable objects, until they implement a way for durable objects to move locations dynamically this wont happen. When this feature does become available, the database will be dynamically moved based on where the most of your traffic originates from.

Alllllllllllllll the way at the bottom of this page is where you can find that this feature is planned for Durable Objects in the future, and then D1 should follow not long after that:
https://developers.cloudflare.com/durable-objects/reference/data-location

Noitidart•9/25/24, 12:45 PM

Thanks very much this was very helpful!

Noitidart•9/25/24, 12:49 PM

Relocation though isn't what I would want. As I have users everywhere. Just primary should be considered different depending on user ID. Is thus a possibility with f1 in future?

NNoitidart Relocation though isn't what I would want. As I have users everywhere. Just prim...

lambrospetrou•9/25/24, 1:28 PM

That's the dynamic bindings feature several folks mentioned already above, where you can have different DBs for each of your users. D1 didn't announce anything about it yet.

NNoitidart Relocation though isn't what I would want. As I have users everywhere. Just prim...

boop•9/25/24, 2:21 PM

As @lambrospetrou said, when you create the DB for your user you can set the primary location of that database. Once it's set, you cannot change it

Dynamic bindings is how you'll interact with the database, and you wont be able to supply any location hint at query time, since the database has already been created with your specified parameters. Your write query will travel back to the primary, and your read queries will come from any replicas (once this feature is available) if the data is hot

boop•9/25/24, 2:22 PM

If it operates similarly to how DOs operate, then you won't even have to think about supplying location hints, since the database will be created in the closest region to your users request anyway

Noitidart•9/25/24, 5:24 PM

Thanks guys for this brainstorm. Multiple independent DB's isn't something I can do. I needed one DB, with read and write replicas. Where the write happens also to the nearest DB but would eventually propagate to all others.

TomK•9/26/24, 6:08 AM

I apologize for the interruption, but I am facing a problem and hope to find some solutions. The issue is "D1_ERROR: Network connection lost". I have searched the internet and discord and found that many people have reported this issue but have not found any effective solutions. This problem does not always occur, but sporadically. In the past 24 hours, my worker has only handled 4.5K requests, which is relatively low. So I want to understand whether this is because of my incorrect usage, or due to the instability of the D1 system itself?

Robin GenzOP•9/26/24, 7:21 AM

Hey

Is there an easy way to seed the local d1 database with data using typescript? I know that I can create a sql file which is then executed by wrangler. But I have a custom typescript file that is called by tsx and uses drizzle + fakerjs and I can't manage to connect to the database. Is there a possibility?

oof2win2•9/26/24, 3:23 PM

By the end of 2024, D1 will support automatic read replication for scalability and low-latency access around the world

hell yeah

Ooof2win2 > By the end of 2024, D1 will support automatic read replication for scalability...

Zack•9/26/24, 3:24 PM

Wait, where did you see this

oof2win2•9/26/24, 3:25 PM

https://blog.cloudflare.com/sqlite-in-durable-objects

The Cloudflare Blog

Zero-latency SQLite storage in every Durable Object

Traditional cloud storage is inherently slow because it is accessed over a network and must synchronize many clients. But what if we could instead put your application code deep into the storage layer, such that your code runs where the data is stored? Durable Objects with SQLite do just that.

Zack•9/26/24, 3:25 PM

Oh nice, hadn't read that one yet

oof2win2•9/26/24, 3:29 PM

honestly what would be nice would be libsql addon support

oof2win2•9/26/24, 3:29 PM

like altering columns and renaming tables and stuff

Ben•9/26/24, 4:25 PM

Yea seems like a great fit for this, could also maybe enable embedded replicas and maybe libSQL server.

Ooof2win2 https://blog.cloudflare.com/sqlite-in-durable-objects

marko.333•9/26/24, 4:50 PM

This is stunning.
I assume (hope) that the DO sqlite is functionally equivalent to D1 (including FTS5 and Json modules)?

BBen Yea seems like a great fit for this, could also maybe enable embedded replicas a...

oof2win2•9/26/24, 5:07 PM

that's probably out of scope and libsql server would mean that there would need to be a connection url - which is not the way cloudflare has it sorted out now

oof2win2•9/26/24, 5:07 PM

just adds overhead, instead an easier way is to make a separate worker that connects via the libsql c/rust client internally

Ooof2win2 that's probably out of scope and libsql server would mean that there would need ...

Ben•9/26/24, 6:08 PM

sorry I thought I was in the durable object channel xD - there are now two sqlite services

Mmarko.333 This is stunning. I assume (hope) that the DO sqlite is functionally equivalent ...

Acе•9/27/24, 12:02 AM

It’s the same stuff that D1 is built on apparently, so those should be there. You could even build your own D1 service with this

DDeep @qo7ems @Acе @Isaac | AS213339 thanks everyone! i have a unique use case for D...

stibbs•9/27/24, 6:05 AM

Is it even possible to specify the region for the db? I assumed if you had regulated data sovereignty concerns D1 was simply not an option? The closest control I'm aware of on this is providing a "location hint" but the docs say

Warning: Providing a location hint does not guarantee that D1 runs in your preferred location. Instead, it will run in the nearest possible location (by latency) to your preference.

Warning: Providing a location hint does not guarantee that D1 runs in your preferred location. Instead, it will run in the nearest possible location (by latency) to your preference.

(source)

Jikyu•9/27/24, 12:58 PM

Would it be possible to use drizzle / run drizzle migrations against the new DO sql database, like we can for D1?

Hey 👋🏻 Quick question: Is there an easy way to find the most „expensive“ (most reads/writes) queri

Similar Threads