Our intent is to move to row-based pricing (noting that we will NOT be enabling billing just yet) —

Our intent is to move to row-based pricing (noting that we will NOT be enabling billing just yet) — charging on rows read (scanned) and written, and returning both read/write counts for every query so you can optimize/measure, as well as dashboard analytics (+ GraphQL API) for longer-term / higher-level analysis.

We think rows are MUCH easier to reason about (vs. bytes) and incentivize folks to use indexes for both performance and cost benefits, which is win-win.

kian•8/18/23, 2:49 PM

MMatt Silverlock Our intent is to move to row-based pricing (noting that we will NOT be enabling ...

Unsmart•8/18/23, 2:57 PM

Does this mean that each unit is now going to be priced higher? Or are y'all going to basically eat the cost difference if someone wouldve used multiple units on a single row?

UUnsmart Does this mean that each unit is now going to be priced higher? Or are y'all goi...

Matt SilverlockOP•8/18/23, 3:07 PM

Right now, no intent to change $/unit cost. Just the unit itself is changing. In balance, "95%" of users are reading small rows and writing small rows. Some fraction of users writing 100M 1MB rows every day isn't going to change the dynamics.

IndependentCreator•8/18/23, 3:37 PM

Hi, I'm excited to try D1 in an upcoming project and I saw the disclaimer

While in the Open Alpha period, there is a possibility of breaking changes. The Alpha is meant for testing purposes and using it for production traffic is not recommended.

While in the Open Alpha period, there is a possibility of breaking changes. The Alpha is meant for testing purposes and using it for production traffic is not recommended.

By breaking changes do you mean the API is not yet stable? Is there also a risk of data loss?

Unsmart•8/18/23, 3:51 PM

Its API stability yes, dont think anyone has lost data

MMatt Silverlock Our intent is to move to row-based pricing (noting that we will NOT be enabling ...

Hello, I’m Allie!•8/18/23, 5:24 PM

How would that work with Indexes then? Since a row read from an index would be priced the same as a row read from the table itself? Or am I missing something?

Unsmart•8/18/23, 5:25 PM

If you dont have an index it has to scan many rows from the table not just the one you want afaik

HHello, I’m Allie!How would that work with Indexes then? Since a row read from an index would be p...

Matt SilverlockOP•8/18/23, 5:25 PM

A read that relies on an index (you don't read data from the index itself) is the index read + the table row(s)

UUnsmart If you dont have an index it has to scan many rows from the table not just the o...

Hello, I’m Allie!•8/18/23, 5:28 PM

Am I just completely misunderstanding how indexes work then? I thought(at least as a mental model) that an index was a table with a column for the ID of the attached table, plus whatever column being indexed. So searching for foo=bar may still take 1k row reads(since each column value has its own row), but you just wouldn’t be loading the entire row into memory, just the targeted column and the ID

HHello, I’m Allie!Am I just completely misunderstanding how indexes work then? I thought(at least ...

Matt SilverlockOP•8/18/23, 5:30 PM

Has nothing to do with memory at all (that's orthogonal). For example:
- You have an index on a

created_at

created_at

column of a

users

users

table
- You query

SELECT * FROM users WHERE created_at = ?1

SELECT * FROM users WHERE created_at = ?1

- that will use the index to look up the rows in the

users

users

table. This means there is a read on the index, and then a (MUCH SMALLER) read on the

users

users

table as you no longer have to do a full scan for the row(s) matching your

created_at

created_at

filter.
- Without the index, it'd be effectively a full table sort / scan - e.g. a 500,000 row table is going to read 500,000 rows in that case.
- With the index, depending on your query, you could read as few as two rows (one index lookup, one row from the

users

users

table, if your query only matches one row)

MMatt Silverlock Has nothing to do with memory at all (that's orthogonal). For example: - You hav...

Hello, I’m Allie!•8/18/23, 5:33 PM

Oh, so an index is like a Hash Table, rather than a standard table… That makes sense.

Hello, I’m Allie!•8/18/23, 5:34 PM

Also, another question. If I set LIMIT 1 on a query, does it stop scanning once it finds a match, or does it keep going to find all matches, then only return one?

HHello, I’m Allie!Oh, so an index is like a Hash Table, rather than a standard table… That makes s...

Matt SilverlockOP•8/18/23, 5:39 PM

yep, that's pretty much the case for most databases. There's some nuance around actual data structures depending on query engine. In SQLite that's a B-Tree, but Postgres has higher-level concepts (GiST; GiN)

HHello, I’m Allie!Oh, so an index is like a Hash Table, rather than a standard table… That makes s...

Matt SilverlockOP•8/18/23, 5:40 PM

https://use-the-index-luke.com/sql/anatomy

Anatomy of an SQL Index: What is an SQL Index

What does an index look like and how does it work? Learn about the B-tree, why an index doesn’t get unbalanced but can still become slow.

HHello, I’m Allie!Also, another question. If I set LIMIT 1 on a query, does it stop scanning once ...

Matt SilverlockOP•8/18/23, 5:42 PM

For a naive

SELECT * FROM table LIMIT N

SELECT * FROM table LIMIT N

- yes. But if you have a filter like a

WHERE

WHERE

HAVING

HAVING

clause, then you have to scan more rows to match the filter, since your table isn't necessarily ordered.

HHello, I’m Allie!Also, another question. If I set LIMIT 1 on a query, does it stop scanning once ...

Matt SilverlockOP•8/18/23, 5:43 PM

e.g.

➜  wrangler d1 execute db-wnam --command "SELECT * FROM [Order] WHERE ShipRegion LIKE '%Western%' LIMIT 10" --json | jq '.[].meta.rows_read'
18 <-- we had to scan 18 rows to find 10 that matched our filter

➜  wrangler d1 execute db-wnam --command "SELECT * FROM [Order] WHERE ShipRegion LIKE '%Western%' LIMIT 10" --json | jq '.[].meta.rows_read'
18 <-- we had to scan 18 rows to find 10 that matched our filter

Unsmart•8/18/23, 5:44 PM

hmm that seems weird I feel like order shouldnt matter it would just scan and the first 10 matches return and doesnt look at the rest only if you add an order by

UUnsmart hmm that seems weird I feel like order shouldnt matter it would just scan and th...

Matt SilverlockOP•8/18/23, 5:45 PM

There's no

ORDER

ORDER

in that query. But it has to scan the table somehow - usually a binary search. It's going to keep searching (scanning) until it has matched all possible records, or hit the

LIMIT

LIMIT

Unsmart•8/18/23, 5:47 PM

Yeah there needs to be some order but I think we are both thinking the same thing. I originally read that as like it will read the entire table even if you already found a number of records matching the limit but it will find only the first rows up to limit but it can possibly scan the entire table if there are not enough to fill up the limit

Unsmart•8/18/23, 5:49 PM

But like if you have 100 rows with 50% being value "1" and 50% being "2" and you do a where value = "1" if you get insanely lucky with the scan it could read as low as 50 rows and very bad luck would read all 100. something like that

UUnsmart Yeah there needs to be some order but I think we are both thinking the same thin...

Matt SilverlockOP•8/18/23, 5:49 PM

yep, exactly. It could scan the whole table and only find 5 results for

LIMIT 10

LIMIT 10

UUnsmart But like if you have 100 rows with 50% being value "1" and 50% being "2" and you...

Matt SilverlockOP•8/18/23, 5:51 PM

Yep! Just Binary Search Things

UUnsmart hmm that seems weird I feel like order shouldnt matter it would just scan and th...

Matt SilverlockOP•8/18/23, 5:54 PM

To add: you can have a sorted index (in D1 it doesn't matter which direction; the query engine can just scan the other way) which will further improve index efficiency if you are doing a lot of

ORDER BY <some_date_column>

ORDER BY <some_date_column>

queries. I probably need to add this to the docs.

MMatt Silverlock To add: you _can_ have a sorted index (in D1 it doesn't matter which direction; ...

Matt SilverlockOP•8/18/23, 5:55 PM

some reading if you're interested: https://stackoverflow.com/a/57082926/556573

Stack Overflow

Sort order for sqlite index

There is sort order when creating index for sqlite.

https://sqlite.org/lang_createindex.html
Each column name or expression can be followed by one of the "ASC" or "DESC" keywords to indicate ...

Unsmart•8/18/23, 5:57 PM

yeah thanks

MMatt Silverlock e.g. ``` ➜ wrangler d1 execute db-wnam --command "SELECT * FROM [Order] WHERE S...

Hello, I’m Allie!•8/18/23, 6:39 PM

Thanks for clarifying!

chientrm.com•8/18/23, 8:12 PM

do we need more than

SQL LIKE

SQL LIKE

MMatt Silverlock D1 now returns `rows_written` and `rows_read` in the metadata for every query. D...

Matt SilverlockOP•8/18/23, 9:53 PM

https://developers.cloudflare.com/d1/changelog/#row-count-now-returned-per-query

Changelog · Cloudflare D1 docs

D1 now returns a count of rows_written and rows_read for every query executed, allowing you to assess the cost of query for both pricing and index …

PPaganMuffin If I understand it correctly. e.g. ```SELECT * FROM table WHERE index_column =...

Matt SilverlockOP•8/18/23, 10:54 PM

I don’t know where you get 51 from? Your second example doesn’t seem right either. The actual behavior depends on if the index is ordered as well.

MMatt Silverlock D1 now returns `rows_written` and `rows_read` in the metadata for every query. D...

TigersWay•8/18/23, 11:02 PM

I just had a quick check to read those new

rows_written

rows_written

and

rows_read

rows_read

and I am kinda surprised to find written rows in a SELECT query

TigersWay•8/18/23, 11:03 PM

And way more read than there is rows!

TTigersWay I just had a quick check to read those new `rows_written` and `rows_read` and I ...

Matt SilverlockOP•8/18/23, 11:23 PM

What is your query? A SELECT isn’t guaranteed to be not write rows.

Unsmart•8/18/23, 11:28 PM

sql is confusing

CCole Xemi Is this going to be visible in the dashboard eventually?

Matt SilverlockOP•8/19/23, 12:26 AM

Yes. Within the next week or two

Hello, I’m Allie!•8/19/23, 7:32 AM

D1 Pricing

Pricing · Cloudflare D1 docs

While in public Alpha, D1 is currently free to use on all Workers plans. Refer to the our recent announcement ) for more information.

jahir9991•8/19/23, 7:59 AM

In production pages my get is working but writhing on db is not working is says no authorization can anyone aware of that i am using Sveltekit and drizzle and all binding are okay as i can get the data , my local dev is also working fine

kian•8/19/23, 8:06 AM

Do you have an example of a query that is failing?

jahir9991•8/19/23, 8:26 AM

Put and post is failing

jahir9991•8/19/23, 8:26 AM

It says not authorized

Kkian Do you have an example of a query that is failing?

jahir9991•8/19/23, 8:28 AM

Do i need to set any env variable rather than d1 binding

Kkian Do you have an example of a query that is failing?

jahir9991•8/19/23, 8:30 AM

https://staging.jahir.pages.dev/api/skills?q= this is get request which is working fine and serving data from d1

TigersWay•8/19/23, 8:36 AM

Following the apparition of

rows_written

rows_written

and

rows_read

rows_read

in every result, could we imagine some total_rows_writtentotal_rows_written and total_rows_readtotal_rows_read in a "status" request?

Original message was deleted

Matt SilverlockOP•8/19/23, 8:38 AM

Keep in mind D1 won’t charge extra for replication (and it won’t multiply storage) or Time Travel (point in time recovery). $5/mo gets you 50M writes and 25B reads, 30 days Time Travel and storage.

TTigersWay Following the apparition of `rows_written` and `rows_read` in every result, coul...

Matt SilverlockOP•8/19/23, 8:39 AM

Can you expand? All of this will be in the GraphQL analytics API & dashboard

TTigersWay Following the apparition of `rows_written` and `rows_read` in every result, coul...

Matt SilverlockOP•8/19/23, 8:39 AM

e.g. https://developers.cloudflare.com/d1/platform/metrics-analytics/

Metrics and analytics · Cloudflare D1 docs

D1 exposes database analytics that allow you to inspect query volume, response size, and query latency across all and/or each database in your …

TigersWay•8/19/23, 8:40 AM

I guess I'll have to jump in that one! Thanks

Kkian Do you have an example of a query that is failing?

jahir9991•8/19/23, 11:20 AM

a hello @kiannh thanks for your replay , i got the issue , i added transaction in my query !!!!!!!!!!!!!, and its become rude ,
any way to use transaction ?

Our intent is to move to row-based pricing (noting that we will NOT be enabling billing just yet) —

Similar Threads

Similar Threads

Similar Threads