Have you looked at [Cloudflare Pipelines](https://developers.cloudflare.com/pipelines/)? It's design

Have you looked at Cloudflare Pipelines? It's designed for high-speed data ingestion. The data goes into an R2 bucket in time slices that you can batch process.

Sylvain•7/23/25, 1:18 PM

Any plans to allow using read replicas via the D1 REST API?

LLarry Have you looked at [Cloudflare Pipelines](https://developers.cloudflare.com/pipe...

KK•7/23/25, 1:20 PM

I'm also planning to query this data once it's ingested—do you know if R2 provides low enough latency and throughput for efficient querying, or would that require an additional system layered on top?

KKK I'm also planning to query this data once it's ingested—do you know if R2 provid...

LarryOP•7/23/25, 1:32 PM

Yeah, you would need to process the R2 into something you could query. Unless a bunch of raw entries is boiled down into a small number of rows, I don't think D1, DOs, or KV are ideal for your situation. D1 and DOs have a 10GB limit. You could shard with DOs as Silvan suggests but that's complexity. KV is not queriable. All three are probably costly for your use. I don't know anything about it but Cloudflare Analytics might be an option. There's also hyperdrive and 3rd party storage integrations

Cloudflare Docs

Hyperdrive

Hyperdrive is a service that accelerates queries you make to existing databases, making it faster to access your data from across the globe from Cloudflare Workers, irrespective of your users' location.

Cloudflare Docs

3rd Party Integrations

Connect to third-party databases such as Supabase, Turso and PlanetScale)

KK•7/23/25, 2:05 PM

thank you very much @Larry

JJames single thread on D1 and network latency is going to kill you, you're better off ...

TeamDave•7/23/25, 2:57 PM

So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of them in a batch using drizzle, and using some results to build the last query.

Locally, this reduced my time by ~500ms. But when I go to staging & use actual D1, I'm getting a cloudflare error, "D1_ERROR: out of memory: SQLITE_NOMEM". I'm not finding a lot about that error, but it seems like I have gone too aggressive with my query combining?

Is it usually a balancing act trying to get the right number of requests with the right complexity of queries?

KKK thank you very much @Larry

Silvan•7/23/25, 3:05 PM

but yea i agree with insane numbers like that i would recommend planetscale

TTeamDave So my sql is not great, but I got it down from 12 queries to 3. I'm sending 2 of...

James•7/23/25, 3:55 PM

Probably a lot of table scanning w your queries, you should analyze the sql plan for your queries and optimize your indexing.

gruntlord6•7/23/25, 8:29 PM

Does anyone know an accurate way to see your D1 Writes? the billing page doesn't line up at all with the actual data in the d1 dashboard

gruntlord6•7/23/25, 8:30 PM

its saying I have millions more writes then any of my databases account for

gruntlord6•7/23/25, 8:31 PM

Edit: ignore this part, it was just my billing period and I I was reading it wrong

gruntlord6•7/23/25, 9:21 PM

that explains the date but they still don't line up

gruntlord6•7/23/25, 9:24 PM

theres a 50 million write difference between what the dashboard shows and what my bill shows

gruntlord6•7/23/25, 9:26 PM

I also never get my billing notification when I go over but apprently thats a beta so I just assume it never will work

gruntlord6•7/23/25, 9:26 PM

just a bit frustrating to get billed with no way to know whats happening

gruntlord6•7/23/25, 9:36 PM

well I managed to track it down but I can't say I am happy that the notification didn't trigger or that I had to dig so much just to find it

gruntlord6•7/24/25, 5:40 AM

no, it didn't line up with my 2 week history in the actual databases

gruntlord6•7/24/25, 5:40 AM

thanks for that though

Silvan•7/24/25, 12:15 PM

@Community Champion :bibicat:

mrcubi•7/24/25, 12:21 PM

hi i am using cloudflare d1 service and the quote of the service i have 25 billion row read free monthly but i still get bill of that 25 billion monthly can i know why

Mmrcubi hi i am using cloudflare d1 service and the quote of the service i have 25 billi...

Silvan•7/24/25, 12:22 PM

you are on the workers paid plan right

mrcubi•7/24/25, 12:23 PM

Yes

Silvan•7/24/25, 12:23 PM

were you billed 25 dollars?

mrcubi•7/24/25, 12:24 PM

I used 38 billion in July and I got a bill for 38 billion

Silvan•7/24/25, 12:24 PM

so 38 dollar?

mrcubi•7/24/25, 12:24 PM

Yes

mrcubi•7/24/25, 12:25 PM

Silvan•7/24/25, 12:25 PM

?billing

SuperHelpflare•7/24/25, 12:25 PM

Cloudflare's billing system is undergoing some changes/migration. During this period, you may notice extra bills, incorrect bills, or other issues.

To resolve these issues, create a Support ticket by using the following link (you must be logged into your Cloudflare account): https://dash.cloudflare.com/?to=/:account/support

Note that all accounts, even Free plan accounts, can create billing tickets. Tickets are the only way to resolve billing issues; we cannot help you in this Discord.

Silvan•7/24/25, 12:25 PM

best to ask them then

< Sun8ox />•7/24/25, 6:23 PM

Hi, I have a question, how do I use Astro with D1 using bindings? (how do I access database binding) Im searching anywhere for any good documentation but cant find any. Nothing really works.

<< Sun8ox />Hi, I have a question, how do I use Astro with D1 using bindings? (how do I acce...

Silvan•7/24/25, 7:24 PM

you need the cloudflare adapter. rest can be found here https://docs.astro.build/en/guides/integrations-guide/cloudflare/#cloudflare-runtime

Docs

@astrojs/cloudflare

Learn how to use the @astrojs/cloudflare adapter to deploy your Astro project.

SSilvan you need the cloudflare adapter. rest can be found here https://docs.astro.build...

< Sun8ox />•7/24/25, 7:31 PM

thanks

Mmrcubi Yes

1984 Ford Laser•7/25/25, 1:17 PM

Line items on invoices don't show the included quota with your plan. So you actually used 25+37billion rows

1984 Ford Laser•7/25/25, 1:17 PM

Best to check D1 metrics via GraphQL API to dig deeper

George•7/25/25, 6:42 PM

Hi, may I ask for some help?
So I'm using worker with D1 just fine and want to utilize the Read replication feature. I changed my worker code of

SELECT

SELECT

statements to use env.DB.withSession() but get this error:

[UNKNOWN_ERROR] POST /login: Error: Cannot read properties of undefined (reading 'withSession')Internal cause: N/A
Stack: TypeError: Cannot read properties of undefined (reading 'withSession')
at index.js:3007:28
at async dispatch (index.js:929:17)
at async cors2 (index.js:3204:5)
at async dispatch (index.js:929:17)
at async logger2 (index.js:2495:5)
at async dispatch (index.js:929:17)
at async index.js:1773:26

I confirm I've enabled

Read replication

Read replication

in the D1 setting, any clue what's wrong?
btw I'm using hono.dev framework but I don't think it's related. Thanks

James•7/26/25, 1:39 PM

Can you share your full code and a way to reproduce? From that error,

env.DB

env.DB

is undefined.

Pepper•7/26/25, 3:18 PM

I was trying to add rows to my table and I somehow ended up with 6 rows that all have the same value for the primary key, which I didn't think was possible. The issue is that I am now unable to edit any of them. Whenever I make edits in the cloudflare d1 studio they all get reverted when I refresh the table (I am hitting save changes). Deleting the rows also doesn't work, they all reappear when I refresh the table. Does anyone know how I can get rid of these rows? Thanks.

Pepper•7/26/25, 3:28 PM

I think that part of the issue is that when I try to run an sql query selecting by the duplicate primary key it returns no results. But when I select * all of the duplicates appear. So when the d1 studio is trying to make delete requests it isn't able to delete anything because it isn't finding anything

Pepper•7/26/25, 3:34 PM

Since my primary keys were numbers, I was able to fix the problem and delete all of the duplicates by doing:
WHERE primary_key > 499
instead of:
WHERE primary_key = 500

LarryOP•7/26/25, 5:18 PM

I'm still more than simply curious to understand how an impossible thing happened. Can you run,

PRAGMA table_info(table_name)

PRAGMA table_info(table_name)

and show us the output? I'm wondering if the table was originally created with a different primary key and changed your code to something like,

CREATE TABLE IF NOT EXISTS...

CREATE TABLE IF NOT EXISTS...

with the primary key that you now think is the primary key but it actually isn't.

LLarry I'm still more than simply curious to understand how an impossible thing happene...

James•7/26/25, 5:19 PM

definitle not default behavior, messing w the key / rowid / pragma settings could get you there

LLarry I'm still more than simply curious to understand how an impossible thing happene...

Pepper•7/26/25, 6:27 PM

Here is the output (num is what I expected the primary key to be, in my earlier message I wrote primary_key just for the example). It originally happened when I used d1 manager to add rows to the table but didn't enter any values. But I was able to replicate this by doing:
INSERT INTO queue DEFAULT VALUES;
What that query does for me is insert everything as null except for the primary key, which it sets to be equal to the largest primary key in my database.

LarryOP•7/26/25, 7:11 PM

Ohh, I remember that bug in SQLite that allows null primary key values unless you create the table with that column set to

NOT NULL

NOT NULL

. https://sqlite.org/quirks.html#primary_keys_can_sometimes_contain_nulls

LarryOP•7/26/25, 7:20 PM

Wait, I'm still confused...

The value of an INTEGER PRIMARY KEY column must always be a non-NULL integer because the INTEGER PRIMARY KEY is an alias for the ROWID. If you try to insert a NULL into an INTEGER PRIMARY KEY column, SQLite automatically converts the NULL into a unique integer.

Ddandoen if only there was a way – as a work around – to do point in time recovery of a d...

dandoen•7/28/25, 8:33 AM

hi all, are there any cloudflare PMs that can shed some light whether this is something on the roadmap for this year? feels fairly important to be able to export a d1 db without blocking all reads/writes.

Original message was deleted

dandoen•7/28/25, 11:12 AM

thanks nora, just worried to go down this path w/ some of my d1 instances being fairly large (and the upperl imit being 10gb)

Ddandoen thanks nora, just worried to go down this path w/ some of my d1 instances being ...

gruntlord6•7/28/25, 3:39 PM

Reading your data is pretty straightforward so you shouldn't really have any issues with checking the integrity of your copy etc. I prefer the d1 dump but the file lock is a big downside for production data

Ggruntlord6 Reading your data is pretty straightforward so you shouldn't really have any iss...

dandoen•7/28/25, 3:43 PM

yeah i don't really see it being feasible for my use case mentioned earlier in the thread; i want to export it and use it in duckdb to power a few analytics heavy queries. i don't need real time, but something like per hour snapshots is what i'm looking to do. exporting by manually reading every row on every table i have is sure possible but a) costly and b) can't achieve fresh enough analytics (as i suspect the exports in this way would take hour+ when applied to a 10gb size db)

really eager to get some guidance from anyone at CF to see if this is something actually in the works for the remainder of this year. multiple docs mention there's but this d1 channel seems abandoned from PM perspective compared to the other channels

gruntlord6•7/28/25, 3:47 PM

Isn't it the same amount of reads to dump the entire database?

Have you looked at [Cloudflare Pipelines](https://developers.cloudflare.com/pipelines/)? It's design

Similar Threads