Hey, are there any plans to provide the possibility for data analytics. As far as I understand I can

Hey, are there any plans to provide the possibility for data analytics. As far as I understand I can access D1 either via worker or via the proprietary http api. Sure someone can build an own etl solution around this but this requires own development efforts (and you have to take care about change data capture etc. yourself). Also you also have to break out of the cloudflare eco system as there is no analytics optimized data store. A good start would be to have the possiblity to etl d1 / kv etc. to R2 -> iceberg or managed Clickhouse (which is already there for workers analytics).

SStefan Stührmann Hey, are there any plans to provide the possibility for data analytics. As far a...

Max (@rozenmd)•1/28/25, 4:20 PM

change data capture is something we'd like to offer with D1, yeah

Robin Genz•1/28/25, 4:27 PM

Hey

Quick question: Is there any way to increase the 10 GB limit of Cloudflare D1 for my account? At the moment it is not a problem but we are growing very fast and will reach the limit in the next few months.

MMax (@rozenmd)change data capture is something we'd like to offer with D1, yeah

Stefan StührmannOP•1/28/25, 4:31 PM

how would you implement it cdc -> r2 with append operation like S3 got recently or you plan to introduce a stream like Kafka?

Original message was deleted

iLearn•1/28/25, 9:59 PM

Is the hard-limit of 10GB also applicable in case of DO with "SQLite storage" (which is D1) ?

Original message was deleted

iLearn•1/28/25, 9:59 PM

Thanks for prompt and crisp answer.

Original message was deleted

iLearn•1/28/25, 10:00 PM

Agree.

Original message was deleted

iLearn•1/28/25, 10:01 PM

True. But, can be raised to 10GB on request and ofcourse, in GA it will automatically be raised to 10GB.

rufi•1/29/25, 3:20 PM

Say I read from D1 on every request for some config or the like, would this add something like 200ms latency on each one?

Rrufi Say I read from D1 on every request for some config or the like, would this add ...

lambrospetrou•1/29/25, 3:59 PM

it depends where your database is, and where your workers will be. Usual roundtrip latencies apply. If you query from Europe to a database in US 100-150ms is expected yes. Within the same region, like EEUR/WEUR you can get 20-60ms.

gaminski•1/29/25, 4:55 PM

I need help understanding something. On the D1 marketing page it states we can have up to 50,000 databases on the paid plan. I cannot find a way to bind multiple databases to one project. How is that done? I’ve searched this thread and I realise that dynamic binding isn’t yet a thing. Some people mention binding multiple databases to one worker but I’m not sure how that’s done

Devid•1/29/25, 4:57 PM

Hello, Is there any way to migrate aws rds mariadb database to cloudflare database?

Ggaminski I need help understanding something. On the D1 marketing page it states we can h...

lambrospetrou•1/29/25, 5:23 PM

You can have multiple D1 bindings in your

wrangler.toml

wrangler.toml

. Repeat the

[[d1_databases]]

[[d1_databases]]

section for each of your databases. That's defining an array, and each time you use it, it's an item in that array. The newer

wrangler.json

wrangler.json

format makes that a bit easier to really understand though if you are not familiar with TOML syntax.

Example:

[[d1_databases]]
binding = "DB_01"
database_name = "db-01"
database_id = "<unique-ID-for-your-database>"

[[d1_databases]]
binding = "DB_02"
database_name = "db-02"
database_id = "<unique-ID-for-your-database>"

[[d1_databases]]
binding = "DB_01"
database_name = "db-01"
database_id = "<unique-ID-for-your-database>"

[[d1_databases]]
binding = "DB_02"
database_name = "db-02"
database_id = "<unique-ID-for-your-database>"

DDevid Hello, Is there any way to migrate aws rds mariadb database to cloudflare databa...

lambrospetrou•1/29/25, 5:24 PM

Export/SQL dump your existing database and import it in D1: https://developers.cloudflare.com/d1/best-practices/import-export-data/

Llambrospetrou You can have multiple D1 bindings in your `wrangler.toml`. Repeat the `[[d1_data...

gaminski•1/29/25, 5:25 PM

Thank you - that sounds simple! Is there a way to limit access to a db to specific subdomains in the config file or elsewhere on the worker, or does it need to be done at the application level?

Ggaminski Thank you - that sounds simple! Is there a way to limit access to a db to specif...

lambrospetrou•1/29/25, 5:25 PM

The worker that has this binding has full access to the database. If you have code that you don't want to have access in some of your bindings then it should be in a different worker.

Llambrospetrou The worker that has this binding has full access to the database. If you have co...

gaminski•1/29/25, 5:28 PM

Ok thanks. I think im struggling to grasp some fundamental concepts. Can I have multiple workers for one project? Maybe what I’m looking for is one worker per database? I’m trying to allocate one database per user (I’m not expecting thousands of users so I’m happy with a manual solution for now)

Ggaminski Ok thanks. I think im struggling to grasp some fundamental concepts. Can I have ...

lambrospetrou•1/29/25, 5:30 PM

Number of workers and how you allocate each database to users is very different. Your Worker code is the one that has access to the databases, so your code controls which database to access. So you can have for example 10 D1 databases attached to your worker, and then depending on the request context (e.g. user ID, account ID, or whatever your user identifier is) you decide which of those databases to use.

You definitely don't need 1000 worker projects!

gaminski•1/29/25, 5:38 PM

Thank you. Let’s see if I understand correctly: I have one worker with multiple databases. One TOML file with the bindings defined as you explained earlier. Somewhere in my application I keep a record of which database each user should be able to access and route the queries accordingly

Ashkan•1/29/25, 5:40 PM

AAshkan Click to see attachment

Ashkan•1/29/25, 5:40 PM

rufi•1/29/25, 6:49 PM

ok but I assume D1 is geo-distributed. So If my worker uses smart placement, it would reduce latency between the worker and the D1 databases?
Also, if I need a better latency (something like 20 ms tops), should I rather use KV, or D1 backend on durable objects, ...?

Rrufi ok but I assume D1 is geo-distributed. So If my worker uses smart placement, it ...

lambrospetrou•1/29/25, 7:05 PM

D1 is not yet replicated. It's in a single location.
With Worker smart placement, your worker will be running close to the database assuming you have enough traffic (100+ requests per day) for smart placement to do its magic.

If you need 20ms tops from anywhere in the world there isn't much you can do with ANY database. You need to have replicas all over the place. Either you do them yourself, or wait for D1 replication. Workers KV has very different semantics from D1 or DOs.

Llambrospetrou D1 is not yet replicated. It's in a single location. With Worker smart placement...

rufi•1/29/25, 7:14 PM

ah ok interesting thanks a lot. So when I create the D1 database I can also select a location. Most of our users are in NA anyway.
The use case I have is basically I want to "in memory sync" the configuration to be used when some endpoint is called. That configuration has to be updated say every hour but I want to avoid having to redeploy the workers to embed the config in place.

Rrufi ah ok interesting thanks a lot. So when I create the D1 database I can also sele...

lambrospetrou•1/29/25, 7:29 PM

If you rarely update that config and you have many requests needing it, then you can have D1 as the source of truth where you write your config updates, and then also put them in Workers KV. Your worker can read from Workers KV when a stale value (up to 60s) is OK for that case, and go to D1 if you need the latest/freshest configuration.

Using the local CDN Cache API from your worker sometimes give you faster lookups than Workers KV as well, but it also depends on your scenario. It's not clear from what you said above.

This design can give you the speed when you can use cached values and have D1 as a fallback for your endpoints that need the latest.

Ggaminski Ok thanks. I think im struggling to grasp some fundamental concepts. Can I have ...

Larry•1/30/25, 12:24 AM

"one worker per database" sounds exactly like Cloudflare Durable Objects

gaminski•1/30/25, 12:55 AM

Thanks - I didn’t realise Durable Objects had an SQLite component

Ggaminski Thanks - I didn’t realise Durable Objects had an SQLite component

森

森@VerseLink (RW)•1/30/25, 4:26 AM

fairly new additional, very cool concept, i've been cooking up some really cool prototypes that magically takes the burden away from traditional large db (and also writing JS logic + SQL together is magical, very different compare to Postgres or MongoDB)

AAshkan Click to see attachment

Max (@rozenmd)•1/30/25, 9:56 AM

Assuming you're attempting to provide feedback (hard to tell with your use of images), 1.2 ms means 1.2 milliseconds, 5.05k means 5050 (no unit provided)

MMax (@rozenmd)Assuming you're attempting to provide feedback (hard to tell with your use of im...

Ashkan•1/30/25, 3:50 PM

I wanted to say thanks and great job

majestic•1/30/25, 6:33 PM

How do do you guys manage that working in d1 using drizzle to create and alter table which is using commands like generate & migrate.
Im facing an error after generating new schema and migrate it into db it shows table already existed error

Mmajestic How do do you guys manage that working in d1 using drizzle to create and alter t...

森

森@VerseLink (RW)•1/30/25, 6:53 PM

you need to use wrangler migrate

森森@VerseLink (RW)you need to use wrangler migrate

majestic•1/30/25, 7:00 PM

Pls specific, can u show me the documentations or code ?

Mmajestic Pls specific, can u show me the documentations or code ?

森

森@VerseLink (RW)•1/30/25, 7:08 PM

https://developers.cloudflare.com/d1/reference/migrations/

Cloudflare Docs

Migrations · Cloudflare D1 docs

Database migrations are a way of versioning your database. Each migration is stored as an .sql file in your migrations folder. The migrations folder is created in your project directory when you create your first migration. This enables you to store and track changes throughout database development.

森

森@VerseLink (RW)•1/30/25, 7:09 PM

you can then list or apply (no need for create since drizzle handles that for you)

森

森@VerseLink (RW)•1/30/25, 7:09 PM

https://developers.cloudflare.com/workers/wrangler/commands/#d1-migrations-apply

Cloudflare Docs

Commands - Wrangler · Cloudflare Workers docs

Create, develop, and deploy your Cloudflare Workers with Wrangler commands.

DDivinitii Just got first-round funding on a new product. Intend to build using Durable Obj...

森

森@VerseLink (RW)•1/31/25, 12:13 AM

in your case is it important that your db always read up to date data?

森

森@VerseLink (RW)•1/31/25, 12:13 AM

i imagine that using KV would actually be potentially a good option

森

森@VerseLink (RW)•1/31/25, 12:16 AM

alternatively you can create a DO with sqlite that basically is d1, but with built in replication ability

森

森@VerseLink (RW)•1/31/25, 12:16 AM

(which would be a lot more complicated, but i believe it could be done)

森

森@VerseLink (RW)•1/31/25, 1:06 AM

yeh well that's kind of your best bet at the moment, I believe DO /w sqlite is free on storage prior to GA (which i imagine D1 read replica would be available after GA), and with DO you would have better control over how the underlying sql mechnism works

森

森@VerseLink (RW)•1/31/25, 1:07 AM

I would implement all write operations using Workflow (to write all DOs) and read directly on DOs, depends on if how important up to date data is

Mmajestic How do do you guys manage that working in d1 using drizzle to create and alter t...

戴

戴きますReina•1/31/25, 3:40 PM

I already tried using prisma and drizzle for d1 and difficult to migrate, we can't use url since binding in wrangler

Mmajestic How do do you guys manage that working in d1 using drizzle to create and alter t...

戴

戴きますReina•1/31/25, 3:42 PM

My frustration in here
d1-database

戴戴きますReina My frustration in here https://discord.com/channels/595317990191398933/992060581...

森

森@VerseLink (RW)•1/31/25, 5:17 PM

I use drizzle for migration, so far no problems in my test environment

戴戴きますReina I already tried using prisma and drizzle for d1 and difficult to migrate, we can...

James•1/31/25, 5:47 PM

drizzle migrations works great for me with D1 and DO sqlite, i dont even use drizzle for the ORM, just the migrations and schema definition

JJames drizzle migrations works great for me with D1 and DO sqlite, i dont even use dri...

森

森@VerseLink (RW)•1/31/25, 5:52 PM

Same, I use Kysely for building complex query and raw SQL if it’s very simple operations

森

森@VerseLink (RW)•1/31/25, 5:52 PM

But drizzle migration helps alleviate a lot of headaches when maintaining SQL migrations

[[d1_databases]] binding = "DB_01" database_name = "db-01" database_id = "<unique-ID-for-your-database>" [[d1_databases]] binding = "DB_02" database_name = "db-02" database_id = "<unique-ID-for-your-database>"

Hey, are there any plans to provide the possibility for data analytics. As far as I understand I can

Similar Threads