Cloudflare Developers•2y ago

d1 info needs to hit your actual db to get data, looks like it's overloaded

Bbun i already have a r2 + cache setup it works amazingly well. i have a 10ms latency...

Max (@rozenmd)OP•2/2/24, 4:59 PM

as a const it'd be a 0ms latency

bun•2/2/24, 4:59 PM

yeah but that way i dont increase the bundle size

MMax (@rozenmd)d1 info needs to hit your actual db to get data, looks like it's overloaded

neal•2/2/24, 4:59 PM

is there anything I can do?

neal•2/2/24, 5:00 PM

I database doesn't load in the dashboard anymore either

Nneal is there anything I can do?

Max (@rozenmd)OP•2/2/24, 5:07 PM

i'm looking into it, but i would look for ways to shard the db across multiple d1 dbs (while we don't automatically do that for you yet)

MMax (@rozenmd)i'm looking into it, but i would look for ways to shard the db across multiple d...

neal•2/2/24, 5:15 PM

thank you!

Nneal thank you!

Max (@rozenmd)OP•2/2/24, 5:21 PM

looks like it recovered a bit - but yeah its reaching the limits of what a single d1 db can do in terms of read/writes before we add read replication and autoscaling

Max (@rozenmd)OP•2/2/24, 5:21 PM

would recommend sharding manually for now and caching anything and everything you can

neal•2/2/24, 5:21 PM

Ok thanks good to know, is there an easy way to duplicate the database?

Nneal Ok thanks good to know, is there an easy way to duplicate the database?

Max (@rozenmd)OP•2/2/24, 5:26 PM

not yet, we're working on adding db cloning/forking/dumping, but until then you can iterate through

SELECT * FROM sqlite_master;

SELECT * FROM sqlite_master;

or use https://github.com/nora-soderlund/cloudflare-d1-backups

neal•2/2/24, 5:27 PM

Thanks, also do you think workers key-value would be able to handle the traffic better?

Nneal Thanks, also do you think workers key-value would be able to handle the traffic ...

Max (@rozenmd)OP•2/2/24, 5:30 PM

if you're just writing and then reading lots, KV would stand significantly more load, trouble is replication lag on updates

Max (@rozenmd)OP•2/2/24, 5:32 PM

https://developers.cloudflare.com/kv/reference/how-kv-works/

DDavid Renshaw ~~It seems that every time I do `wrangler dev` or `wrangler d1 execute` (regardl...

Max (@rozenmd)OP•2/2/24, 5:34 PM

max(indexed_column)

max(indexed_column)

should save some reads (i think)

DDavid Renshaw my database has ~280k rows, and I'm seeing that many reads show up in my "row me...

Max (@rozenmd)OP•2/2/24, 5:36 PM

the console doesn't run any queries unless you enter them - check out your network tab as you load the page

Paul•2/2/24, 5:50 PM

Once it’s GA, is there a limit to the size the database can be?

Chaika•2/2/24, 5:51 PM

will always be a limit, they'd rather you shard into mutiple small dbs then one big one -> d1-database

DDavid Renshaw It would be nice if I had a way to navigate to the "metrics" page without hittin...

Chaika•2/2/24, 5:52 PM

do you not have primary keys or any indexes on your tables? Shouldn't be a full scan just to count all the rows

CChaika do you not have primary keys or any indexes on your tables? Shouldn't be a full ...

Ɽ

ⱤØ₥₳₦•2/2/24, 6:27 PM

If i create an index, if table has let's say 10 mil rows, then if i use index query then what will be the number of row reads?

ⱤⱤØ₥₳₦If i create an index, if table has let's say 10 mil rows, then if i use index qu...

Ɽ

ⱤØ₥₳₦•2/2/24, 6:28 PM

What this means

Screenshot_2024-02-02-23-57-40-61_40deb401b9ffe8e1df2f1cc5ba480b12.jpg

Max (@rozenmd)OP•2/2/24, 6:30 PM

I've been working on analytics for D1's queries on the side, starting with wrangler, you'll be able to find your slowest or most expensive queries by rowsRead/rowsWritten

An extremely early preview build of it is available here if you're curious: https://github.com/cloudflare/workers-sdk/pull/4909

Today it uses averages, but on Monday I should have it using sums, so it'll be sorted by queries that run most often.

GitHub

[D1] teach wrangler how to fetch insights about D1's queries by roz...

Experimental feature, WIP.

ⱤⱤØ₥₳₦If i create an index, if table has let's say 10 mil rows, then if i use index qu...

Matt Silverlock•2/2/24, 7:54 PM

It depends entirely on the index and the query!

Natan Parmigiano•2/2/24, 8:04 PM

Hey!
Is there an estimate of what should be the expected per-db storage limit after the beta is over?

2GB would be way too low for any production application, and i understand it will probably be increased, but i'd like to know how high is it expected to go, as i know that the intended use case is "one db per user" but we expected to use it as a normal database.

Natan Parmigiano•2/2/24, 8:09 PM

is there any way of "dynamically binding" a D1 database?
I cant see a way of applying the idea of using D1 in a multitenant environment without needing to manage a worker deployment for every customer..

NNatan Parmigiano Hey! Is there an estimate of what should be the expected per-db storage limit af...

Cyb3r-Jak3•2/2/24, 8:15 PM

See d1-database.

Discord

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

NNatan Parmigiano is there any way of "dynamically binding" a D1 database? I cant see a way of app...

Cyb3r-Jak3•2/2/24, 8:15 PM

Not yet but it something planned

CCyb3r-Jak3 See https://discord.com/channels/595317990191398933/992060581832032316/115723945...

Natan Parmigiano•2/2/24, 8:17 PM

That answers my question!

MMatt Silverlock Double digits makes sense. 100GB _per database_ is just a very different thing. ...

Vignesh•2/3/24, 7:32 AM

For this to happen will Cloudflare does horizontal scaling automatically in the background or are we supposed to do it manually?

MMatt Silverlock It depends entirely on the index and the query!

Ɽ

ⱤØ₥₳₦•2/3/24, 7:51 AM

Like if there is a table with millions of rows and that has timestamp column and i use that for index, then?

ⱤⱤØ₥₳₦Like if there is a table with millions of rows and that has timestamp column and...

Matt Silverlock•2/3/24, 8:47 AM

Then the rows scanned will depend on how the query uses the timestamp and the index. I strongly suggest reading https://developers.cloudflare.com/d1/how-to/using-indexes/

Cloudflare Docs

Use indexes · Cloudflare D1 docs

Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) …