How many requests are you doing? It could simply be you are doing too many queries for the database

How many requests are you doing? It could simply be you are doing too many queries for the database to handle.

UUnsmart How many requests are you doing? It could simply be you are doing too many queri...

neal•2/2/24, 4:05 PM

It was about 500 requests per second during the traffic spike. But now the database appears to be down and all requests are hanging. I may need to migrate to a different database but I can't currently access the data

UnsmartOP•2/2/24, 4:06 PM

Ah yeah 500rps is very upper limits from what I have heard. Fwiw the app works fine for me rn most of the time only some queries fail

neal•2/2/24, 4:07 PM

The cached queries are still working, but anything that's not cached is failing because it needs to access the database unfortunately

Chaika•2/2/24, 4:07 PM

Using beta, not alpha DB right?

Chaika•2/2/24, 4:08 PM

can check via npx wrangler d1 info <db-name>npx wrangler d1 info <db-name>

neal•2/2/24, 4:08 PM

Yes using D1 that says it's in open beta

Chaika•2/2/24, 4:08 PM

there's still two different types of databases. If you made it a while ago, it may be alpha. Alpha had a lot worse performance

Chaika•2/2/24, 4:08 PM

500 rps is still pretty high though

CChaika can check via `npx wrangler d1 info <db-name>`

neal•2/2/24, 4:09 PM

It looks like this also doesn't respond

neal•2/2/24, 4:10 PM

Screen_Shot_2024-02-02_at_11.10.01_AM.png

neal•2/2/24, 4:11 PM

I created the database last month, so I believe it should be the beta version

Nneal I created the database last month, so I believe it should be the beta version

Chaika•2/2/24, 4:27 PM

Yea iirc they disabled new alpha dbs a few months ago
Under Workers & Pages -> D1, the database ID would be helpful (or you could just copy it from that error, it's the one at the end there hyphen seperated. Your Account ID would also be useful. It's under Workers & Pges overview or the /accounts/<accountid> in that error. I would OCR them but it's going to mess them up somewhere. They aren't private or anything but would let the D1 Team look into the DB

neal•2/2/24, 4:39 PM

My account id is "62cafb6b9c2ebd2e6970091b9ef0b6a8"

neal•2/2/24, 4:39 PM

Thanks for your help!

neal•2/2/24, 4:42 PM

Also database id is "eb589e60-ed90-4777-9685-f73fad7daaa3"

MMax (@rozenmd)make it a const in your worker if it never changes

bun•2/2/24, 4:47 PM

const you mean just create it as a variable?

Bbun const you mean just create it as a variable?

Max (@rozenmd)•2/2/24, 4:57 PM

yeah

bun•2/2/24, 4:58 PM

i already have a r2 + cache setup it works amazingly well. i have a 10ms latency on retrieving the cache of my 1 mb JSON file

Nneal Click to see attachment

Max (@rozenmd)•2/2/24, 4:58 PM

d1 info needs to hit your actual db to get data, looks like it's overloaded

Bbun i already have a r2 + cache setup it works amazingly well. i have a 10ms latency...

Max (@rozenmd)•2/2/24, 4:59 PM

as a const it'd be a 0ms latency

bun•2/2/24, 4:59 PM

yeah but that way i dont increase the bundle size

MMax (@rozenmd)d1 info needs to hit your actual db to get data, looks like it's overloaded

neal•2/2/24, 4:59 PM

is there anything I can do?

neal•2/2/24, 5:00 PM

I database doesn't load in the dashboard anymore either

Nneal is there anything I can do?

Max (@rozenmd)•2/2/24, 5:07 PM

i'm looking into it, but i would look for ways to shard the db across multiple d1 dbs (while we don't automatically do that for you yet)

MMax (@rozenmd)i'm looking into it, but i would look for ways to shard the db across multiple d...

neal•2/2/24, 5:15 PM

thank you!

Nneal thank you!

Max (@rozenmd)•2/2/24, 5:21 PM

looks like it recovered a bit - but yeah its reaching the limits of what a single d1 db can do in terms of read/writes before we add read replication and autoscaling

Max (@rozenmd)•2/2/24, 5:21 PM

would recommend sharding manually for now and caching anything and everything you can

neal•2/2/24, 5:21 PM

Ok thanks good to know, is there an easy way to duplicate the database?

Nneal Ok thanks good to know, is there an easy way to duplicate the database?

Max (@rozenmd)•2/2/24, 5:26 PM

not yet, we're working on adding db cloning/forking/dumping, but until then you can iterate through SELECT * FROM sqlite_master;SELECT * FROM sqlite_master; or use https://github.com/nora-soderlund/cloudflare-d1-backups

neal•2/2/24, 5:27 PM

Thanks, also do you think workers key-value would be able to handle the traffic better?

Nneal Thanks, also do you think workers key-value would be able to handle the traffic ...

Max (@rozenmd)•2/2/24, 5:30 PM

if you're just writing and then reading lots, KV would stand significantly more load, trouble is replication lag on updates

Max (@rozenmd)•2/2/24, 5:32 PM

https://developers.cloudflare.com/kv/reference/how-kv-works/

DDavid Renshaw ~~It seems that every time I do `wrangler dev` or `wrangler d1 execute` (regardl...

Max (@rozenmd)•2/2/24, 5:34 PM

max(indexed_column)max(indexed_column) should save some reads (i think)

DDavid Renshaw my database has ~280k rows, and I'm seeing that many reads show up in my "row me...

Max (@rozenmd)•2/2/24, 5:36 PM

the console doesn't run any queries unless you enter them - check out your network tab as you load the page

Paul•2/2/24, 5:50 PM

Once it’s GA, is there a limit to the size the database can be?

Chaika•2/2/24, 5:51 PM

will always be a limit, they'd rather you shard into mutiple small dbs then one big one -> d1-database

DDavid Renshaw It would be nice if I had a way to navigate to the "metrics" page without hittin...

Chaika•2/2/24, 5:52 PM

do you not have primary keys or any indexes on your tables? Shouldn't be a full scan just to count all the rows

CChaika do you not have primary keys or any indexes on your tables? Shouldn't be a full ...

Ɽ

ⱤØ₥₳₦•2/2/24, 6:27 PM

If i create an index, if table has let's say 10 mil rows, then if i use index query then what will be the number of row reads?

ⱤⱤØ₥₳₦If i create an index, if table has let's say 10 mil rows, then if i use index qu...

Ɽ

ⱤØ₥₳₦•2/2/24, 6:28 PM

What this means

Screenshot_2024-02-02-23-57-40-61_40deb401b9ffe8e1df2f1cc5ba480b12.jpg

Max (@rozenmd)•2/2/24, 6:30 PM

I've been working on analytics for D1's queries on the side, starting with wrangler, you'll be able to find your slowest or most expensive queries by rowsRead/rowsWritten

An extremely early preview build of it is available here if you're curious: https://github.com/cloudflare/workers-sdk/pull/4909

Today it uses averages, but on Monday I should have it using sums, so it'll be sorted by queries that run most often.

GitHub

[D1] teach wrangler how to fetch insights about D1's queries by roz...

Experimental feature, WIP.

ⱤⱤØ₥₳₦If i create an index, if table has let's say 10 mil rows, then if i use index qu...

Matt Silverlock•2/2/24, 7:54 PM

It depends entirely on the index and the query!

Natan Parmigiano•2/2/24, 8:04 PM

Hey!
Is there an estimate of what should be the expected per-db storage limit after the beta is over?

2GB would be way too low for any production application, and i understand it will probably be increased, but i'd like to know how high is it expected to go, as i know that the intended use case is "one db per user" but we expected to use it as a normal database.

How many requests are you doing? It could simply be you are doing too many queries for the database

Similar Threads

Similar Threads

Similar Threads