Hi! I'm using the D1 database for a game I'm working on https://twitter.com/nealagarwal/status/17527

Hi! I'm using the D1 database for a game I'm working on https://twitter.com/nealagarwal/status/1752716375255101440

The database appears to be down after a large traffic spike. Is there anyone that could help debug?

Neal Agarwal (@nealagarwal) on X

The first version of Infinite Craft, an endless crafting game, is out now!

https://t.co/tb51ejeqsK

Twitter

•

1/31/24, 3:32 PM

Nneal Hi! I'm using the D1 database for a game I'm working on https://twitter.com/neal...

Unsmart•2/2/24, 3:59 PM

How many requests are you doing? It could simply be you are doing too many queries for the database to handle.

UUnsmart How many requests are you doing? It could simply be you are doing too many queri...

nealOP•2/2/24, 4:05 PM

It was about 500 requests per second during the traffic spike. But now the database appears to be down and all requests are hanging. I may need to migrate to a different database but I can't currently access the data

Unsmart•2/2/24, 4:06 PM

Ah yeah 500rps is very upper limits from what I have heard. Fwiw the app works fine for me rn most of the time only some queries fail

nealOP•2/2/24, 4:07 PM

The cached queries are still working, but anything that's not cached is failing because it needs to access the database unfortunately

Chaika•2/2/24, 4:07 PM

Using beta, not alpha DB right?

Chaika•2/2/24, 4:08 PM

can check via npx wrangler d1 info <db-name>npx wrangler d1 info <db-name>

nealOP•2/2/24, 4:08 PM

Yes using D1 that says it's in open beta

Chaika•2/2/24, 4:08 PM

there's still two different types of databases. If you made it a while ago, it may be alpha. Alpha had a lot worse performance

Chaika•2/2/24, 4:08 PM

500 rps is still pretty high though

CChaika can check via `npx wrangler d1 info <db-name>`

nealOP•2/2/24, 4:09 PM

It looks like this also doesn't respond

nealOP•2/2/24, 4:10 PM

Screen_Shot_2024-02-02_at_11.10.01_AM.png

nealOP•2/2/24, 4:11 PM

I created the database last month, so I believe it should be the beta version

Nneal I created the database last month, so I believe it should be the beta version

Chaika•2/2/24, 4:27 PM

Yea iirc they disabled new alpha dbs a few months ago
Under Workers & Pages -> D1, the database ID would be helpful (or you could just copy it from that error, it's the one at the end there hyphen seperated. Your Account ID would also be useful. It's under Workers & Pges overview or the /accounts/<accountid> in that error. I would OCR them but it's going to mess them up somewhere. They aren't private or anything but would let the D1 Team look into the DB

nealOP•2/2/24, 4:39 PM

My account id is "62cafb6b9c2ebd2e6970091b9ef0b6a8"

nealOP•2/2/24, 4:39 PM

Thanks for your help!

nealOP•2/2/24, 4:42 PM

Also database id is "eb589e60-ed90-4777-9685-f73fad7daaa3"

MMax (@rozenmd)make it a const in your worker if it never changes

bun•2/2/24, 4:47 PM

const you mean just create it as a variable?

Bbun const you mean just create it as a variable?

Max (@rozenmd)•2/2/24, 4:57 PM

yeah

bun•2/2/24, 4:58 PM

i already have a r2 + cache setup it works amazingly well. i have a 10ms latency on retrieving the cache of my 1 mb JSON file

Nneal Click to see attachment

Max (@rozenmd)•2/2/24, 4:58 PM

d1 info needs to hit your actual db to get data, looks like it's overloaded

Bbun i already have a r2 + cache setup it works amazingly well. i have a 10ms latency...

Max (@rozenmd)•2/2/24, 4:59 PM

as a const it'd be a 0ms latency

bun•2/2/24, 4:59 PM

yeah but that way i dont increase the bundle size

MMax (@rozenmd)d1 info needs to hit your actual db to get data, looks like it's overloaded

nealOP•2/2/24, 4:59 PM

is there anything I can do?

nealOP•2/2/24, 5:00 PM

I database doesn't load in the dashboard anymore either

Nneal is there anything I can do?

Max (@rozenmd)•2/2/24, 5:07 PM

i'm looking into it, but i would look for ways to shard the db across multiple d1 dbs (while we don't automatically do that for you yet)

MMax (@rozenmd)i'm looking into it, but i would look for ways to shard the db across multiple d...

nealOP•2/2/24, 5:15 PM

thank you!

Nneal thank you!

Max (@rozenmd)•2/2/24, 5:21 PM

looks like it recovered a bit - but yeah its reaching the limits of what a single d1 db can do in terms of read/writes before we add read replication and autoscaling

Max (@rozenmd)•2/2/24, 5:21 PM

would recommend sharding manually for now and caching anything and everything you can

nealOP•2/2/24, 5:21 PM

Ok thanks good to know, is there an easy way to duplicate the database?

Nneal Ok thanks good to know, is there an easy way to duplicate the database?

Max (@rozenmd)•2/2/24, 5:26 PM

not yet, we're working on adding db cloning/forking/dumping, but until then you can iterate through

SELECT * FROM sqlite_master;

SELECT * FROM sqlite_master;

or use https://github.com/nora-soderlund/cloudflare-d1-backups

nealOP•2/2/24, 5:27 PM

Thanks, also do you think workers key-value would be able to handle the traffic better?

Nneal Thanks, also do you think workers key-value would be able to handle the traffic ...

Max (@rozenmd)•2/2/24, 5:30 PM

if you're just writing and then reading lots, KV would stand significantly more load, trouble is replication lag on updates

Max (@rozenmd)•2/2/24, 5:32 PM

https://developers.cloudflare.com/kv/reference/how-kv-works/

DDavid Renshaw ~~It seems that every time I do `wrangler dev` or `wrangler d1 execute` (regardl...

Max (@rozenmd)•2/2/24, 5:34 PM

max(indexed_column)

max(indexed_column)

should save some reads (i think)

DDavid Renshaw my database has ~280k rows, and I'm seeing that many reads show up in my "row me...

Max (@rozenmd)•2/2/24, 5:36 PM

the console doesn't run any queries unless you enter them - check out your network tab as you load the page

Paul•2/2/24, 5:50 PM

Once it’s GA, is there a limit to the size the database can be?

Chaika•2/2/24, 5:51 PM

will always be a limit, they'd rather you shard into mutiple small dbs then one big one -> d1-database

DDavid Renshaw It would be nice if I had a way to navigate to the "metrics" page without hittin...

Chaika•2/2/24, 5:52 PM

do you not have primary keys or any indexes on your tables? Shouldn't be a full scan just to count all the rows

CChaika do you not have primary keys or any indexes on your tables? Shouldn't be a full ...

Ɽ

ⱤØ₥₳₦•2/2/24, 6:27 PM

If i create an index, if table has let's say 10 mil rows, then if i use index query then what will be the number of row reads?

ⱤⱤØ₥₳₦If i create an index, if table has let's say 10 mil rows, then if i use index qu...

Ɽ

ⱤØ₥₳₦•2/2/24, 6:28 PM

What this means

Screenshot_2024-02-02-23-57-40-61_40deb401b9ffe8e1df2f1cc5ba480b12.jpg

Max (@rozenmd)•2/2/24, 6:30 PM

I've been working on analytics for D1's queries on the side, starting with wrangler, you'll be able to find your slowest or most expensive queries by rowsRead/rowsWritten

An extremely early preview build of it is available here if you're curious: https://github.com/cloudflare/workers-sdk/pull/4909

Today it uses averages, but on Monday I should have it using sums, so it'll be sorted by queries that run most often.

GitHub

[D1] teach wrangler how to fetch insights about D1's queries by roz...

Experimental feature, WIP.

ⱤⱤØ₥₳₦If i create an index, if table has let's say 10 mil rows, then if i use index qu...

Matt Silverlock•2/2/24, 7:54 PM

It depends entirely on the index and the query!

Hi! I'm using the D1 database for a game I'm working on https://twitter.com/nealagarwal/status/17527

Similar Threads