Is D1 having an especially hard time currently? - Cloudflare Developers

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

Mozzy

MMozzy Is D1 having an especially hard time currently?

Matt Silverlock

M

Matt Silverlock•5/19/23, 12:17 PM

There's nothing service wide. Can you expand on what you're seeing/the issue you're having?

Mozzy

M

MozzyOP•5/19/23, 12:18 PM

I saw an increase in D1_ERROR 500

Mozzy

M

MozzyOP•5/19/23, 12:22 PM

This is also having a hard time loading

Mozzy

M

MozzyOP•5/19/23, 12:22 PM

668d0e1f-b01a-4048-8796-fdf4530fb192668d0e1f-b01a-4048-8796-fdf4530fb192

Mozzy

M

MozzyOP•5/19/23, 12:23 PM

General issues POSTing to the DB

{
  "message": "D1_ERROR",
  "cause": "Error: Error 9000: something went wrong"
}

{
  "message": "D1_ERROR",
  "cause": "Error: Error 9000: something went wrong"
}

Matt Silverlock

M

Matt Silverlock•5/19/23, 1:02 PM

Big new updates to D1 incoming: https://blog.cloudflare.com/d1-turning-it-up-to-11/

The Cloudflare Blog

D1: We turned it up to 11

We've been heads down iterating on D1, and we've just shipped a major new version that's substantially faster, more reliable, and introduces Time Travel: the ability to restore a D1 database to any point in time.

Mozzy

M

MozzyOP•5/19/23, 1:03 PM

Jason Hostetter

J

Jason Hostetter•5/19/23, 1:33 PM

when we enable global read replication, you won’t have to pay extra for it, nor will replication multiply your storage consumption.

when we enable global read replication, you won’t have to pay extra for it, nor will replication multiply your storage consumption.

Jason Hostetter

J

Jason Hostetter•5/19/23, 1:34 PM

very excited, amazing work

Matt Silverlock

MMatt Silverlock Big new updates to D1 incoming: https://blog.cloudflare.com/d1-turning-it-up-to-...

João Castro

J

João Castro•5/19/23, 1:38 PM

this is reeeeally cool

Mozzy

M

MozzyOP•5/19/23, 1:43 PM

What would be the ideal way to migrate an existing DB to the Beta/experimental version?

Mozzy

MMozzy What would be the ideal way to migrate an existing DB to the Beta/experimental v...

Matt Silverlock

M

Matt Silverlock•5/19/23, 1:45 PM

Backup + download locally (https://developers.cloudflare.com/d1/learning/backups/#downloading-a-backup-locally) -> convert to text format (https://developers.cloudflare.com/d1/learning/importing-data/#converting-sqlite-database-files) -> import (https://developers.cloudflare.com/d1/learning/importing-data/#import-an-existing-database).

Mozzy

M

MozzyOP•5/19/23, 1:47 PM

Is this not correct?
data.sqlite3 is the file received from downloading the backup (but renamed)

Mozzy

MMozzy Is this not correct? data.sqlite3 is the file received from downloading the back...

Matt Silverlock

M

Matt Silverlock•5/19/23, 1:48 PM

Don't open the shell. Just invoke it directly.

Original message was deleted

Matt Silverlock

M

Matt Silverlock•5/19/23, 1:49 PM

https://github.com/cloudflare/cloudflare-docs/pull/9005

Matt Silverlock

MMatt Silverlock Don't open the shell. Just invoke it directly.

Mozzy

M

MozzyOP•5/19/23, 1:51 PM

Ahh, thanks

Matt Silverlock

M

Matt Silverlock•5/19/23, 1:54 PM

https://twitter.com/KentonVarda/status/1659551757796515846?s=20 for those that want to understand a bit more about how our new storage backend works under the hood

Kenton Varda (@KentonVarda)

So I've spent the last ~three months writing a new distributed storage engine based on SQLite and it has been FUN!

https://t.co/NVorwS46x8

Twitter

•

5/19/23, 1:29 PM

Atinux

A

Atinux•5/19/23, 2:01 PM

Congrats on the updates on D1, the https://nuxt-todos-edge.pages.dev/ is much faster now

Before / After the upgrade to experimental backend

CleanShot_2023-05-19_at_15.58.592x.png

CleanShot_2023-05-19_at_15.59.182x.png

Atinux

A

Atinux•5/19/23, 2:03 PM

Note that before I could not choose the primary location (I suspected the US), now it's in Western Europe

Mozzy

M

MozzyOP•5/19/23, 2:13 PM

Is it unrealistic to hope that this will complete? db.sql is about 30,000 lines

Matt Silverlock

MMatt Silverlock https://twitter.com/KentonVarda/status/1659551757796515846?s=20 for those that w...

arbfay

A

arbfay•5/19/23, 2:21 PM

Makes me think of this older blog post (written before any D1): https://ma.rkusa.st/store-sqlite-in-cloudflare-durable-objects
DO are the way to go!

Advany

A

Advany•5/19/23, 2:23 PM

The new D1 seems really cool!

I have list of jsons (one per host). I need to get it before each worker is run.

The average json is 260 bytes (up to 1 mb).

Currently using redis/upstash for fast global replication and reads.

Looking at the latency of d1. Seems like its only 20ms orso.

Would it make sense to move it? My total database is only 100mb orso.

Unsmart

U

Unsmart•5/19/23, 2:24 PM

When its GA sure it could

Mozzy

MMozzy Is it unrealistic to hope that this will complete? db.sql is about 30,000 lines

Mozzy

M

MozzyOP•5/19/23, 2:42 PM

Any way to know if it's actually working or has quietly failed and is refusing to give up?

sourabh.sashank

Ssourabh.sashank Hey @mozzy did you solved this ?

Mozzy

M

MozzyOP•5/19/23, 2:44 PM

Yes: d1-database

sourabh.sashank

Ssourabh.sashank Didn't got that actually can you please explain ?

Mozzy

M

MozzyOP•5/19/23, 2:47 PM

Open cmd inside the folder that has sqlite3.exe -> run sqlite3 file.sqlite3 .dump > db.sqlsqlite3 file.sqlite3 .dump > db.sql

MyZeD

M

MyZeD•5/19/23, 3:06 PM

i like progress in d1.
would be cool if where's a compress and uncompress-function.

Mozzy

MMozzy Is it unrealistic to hope that this will complete? db.sql is about 30,000 lines

Mozzy

M

MozzyOP•5/19/23, 3:07 PM

@elithrar Is it expected to take more than an hour to process ~31K INSERT INTO statements from the .sql file?
Unsure if it'll ever complete

Mozzy

MMozzy @elithrar Is it expected to take more than an hour to process ~31K INSERT INTO s...

Matt Silverlock

M

Matt Silverlock•5/19/23, 3:10 PM

It'll be

wrangler

wrangler

trying to parse your file. We haven't optimized this yet (it's unrelated to D1 / the storage backend). Break it up into ~5k line files.

Mozzy

MMozzy @elithrar Is it expected to take more than an hour to process ~31K INSERT INTO s...

Max (@rozenmd)

M

Max (@rozenmd)•5/19/23, 3:11 PM

try pasting it into the console - the ui will lag, but insert should go through

Matt Silverlock

MMatt Silverlock It'll be `wrangler` trying to parse your file. We haven't optimized this yet (it...

Matt Silverlock

M

Matt Silverlock•5/19/23, 3:11 PM

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

Mozzy

M

MozzyOP•5/19/23, 3:11 PM

Thanks, I'll try both

Mozzy

M

MozzyOP•5/19/23, 3:29 PM

Splitting the file has led to success

SmallShen

S

SmallShen•5/19/23, 3:58 PM

I want to know the performance difference between json query and native query, are there any benchmarks?

SmallShen

S

SmallShen•5/19/23, 4:00 PM

It looks great to make d1 a “schema-less” database, but performance is important

Mozzy

M

MozzyOP•5/19/23, 4:02 PM

Sadly seeing very random Error: D1_ERRORError: D1_ERRORs
Sometimes it responds within 150ms, other times it errors

Mozzy

M

MozzyOP•5/19/23, 4:04 PM

Did anything regarding the non-experimental D1 change today as well?
This is the file it's failing in: https://github.com/leonlarsson/bfstats-api/blob/main/handlers/cacheD1.ts

Mozzy

M

MozzyOP•5/19/23, 4:05 PM

Specifically started having issues around this time

Mozzy

MMozzy Sadly seeing very random `Error: D1_ERROR`s Sometimes it responds within 150ms, ...

Matt Silverlock

M

Matt Silverlock•5/19/23, 4:12 PM

Nothing regarding the original alpha changed (at all).

Mozzy

MMozzy Did anything regarding the non-experimental D1 change today as well? This is the...

Matt Silverlock

M

Matt Silverlock•5/19/23, 4:13 PM

If your database has grown, then it's likely you're just running into the limitations of the original alpha

SmallShen

SSmallShen It looks great to make d1 a “schema-less” database, but performance is important

Matt Silverlock

M

Matt Silverlock•5/19/23, 4:14 PM

It's going to really depend on what you're querying and what the shape of the objects you're using is. You also need to take into account the benefits of doing this in your database vs. in your application.

sourabh.sashank

Ssourabh.sashank "D1Cause-GetAllGuruData","value":"Error: Network connection lost. after moving D...

Matt Silverlock

M

Matt Silverlock•5/19/23, 4:19 PM

Can you share more information please? Specifically: your database ID, the query itself (with any PII redacted), and whether this is repeatable?

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms