Has nothing to do with memory at all (that's orthogonal). For example: - You have an index on a `cre

Has nothing to do with memory at all (that's orthogonal). For example:
- You have an index on a

created_at

created_at

column of a

users

users

table
- You query

SELECT * FROM users WHERE created_at = ?1

SELECT * FROM users WHERE created_at = ?1

- that will use the index to look up the rows in the

users

users

table. This means there is a read on the index, and then a (MUCH SMALLER) read on the

users

users

table as you no longer have to do a full scan for the row(s) matching your

created_at

created_at

filter.
- Without the index, it'd be effectively a full table sort / scan - e.g. a 500,000 row table is going to read 500,000 rows in that case.
- With the index, depending on your query, you could read as few as two rows (one index lookup, one row from the

users

users

table, if your query only matches one row)

MMatt Silverlock Has nothing to do with memory at all (that's orthogonal). For example: - You hav...

Hello, I’m Allie!•8/18/23, 5:33 PM

Oh, so an index is like a Hash Table, rather than a standard table… That makes sense.

Hello, I’m Allie!•8/18/23, 5:34 PM

Also, another question. If I set LIMIT 1 on a query, does it stop scanning once it finds a match, or does it keep going to find all matches, then only return one?

HHello, I’m Allie!Oh, so an index is like a Hash Table, rather than a standard table… That makes s...

Matt SilverlockOP•8/18/23, 5:39 PM

yep, that's pretty much the case for most databases. There's some nuance around actual data structures depending on query engine. In SQLite that's a B-Tree, but Postgres has higher-level concepts (GiST; GiN)

HHello, I’m Allie!Oh, so an index is like a Hash Table, rather than a standard table… That makes s...

Matt SilverlockOP•8/18/23, 5:40 PM

https://use-the-index-luke.com/sql/anatomy

Anatomy of an SQL Index: What is an SQL Index

What does an index look like and how does it work? Learn about the B-tree, why an index doesn’t get unbalanced but can still become slow.

HHello, I’m Allie!Also, another question. If I set LIMIT 1 on a query, does it stop scanning once ...

Matt SilverlockOP•8/18/23, 5:42 PM

For a naive

SELECT * FROM table LIMIT N

SELECT * FROM table LIMIT N

- yes. But if you have a filter like a

WHERE

WHERE

HAVING

HAVING

clause, then you have to scan more rows to match the filter, since your table isn't necessarily ordered.

HHello, I’m Allie!Also, another question. If I set LIMIT 1 on a query, does it stop scanning once ...

Matt SilverlockOP•8/18/23, 5:43 PM

e.g.

➜  wrangler d1 execute db-wnam --command "SELECT * FROM [Order] WHERE ShipRegion LIKE '%Western%' LIMIT 10" --json | jq '.[].meta.rows_read'
18 <-- we had to scan 18 rows to find 10 that matched our filter

➜  wrangler d1 execute db-wnam --command "SELECT * FROM [Order] WHERE ShipRegion LIKE '%Western%' LIMIT 10" --json | jq '.[].meta.rows_read'
18 <-- we had to scan 18 rows to find 10 that matched our filter

Unsmart•8/18/23, 5:44 PM

hmm that seems weird I feel like order shouldnt matter it would just scan and the first 10 matches return and doesnt look at the rest only if you add an order by

UUnsmart hmm that seems weird I feel like order shouldnt matter it would just scan and th...

Matt SilverlockOP•8/18/23, 5:45 PM

There's no

ORDER

ORDER

in that query. But it has to scan the table somehow - usually a binary search. It's going to keep searching (scanning) until it has matched all possible records, or hit the

LIMIT

LIMIT

Unsmart•8/18/23, 5:47 PM

Yeah there needs to be some order but I think we are both thinking the same thing. I originally read that as like it will read the entire table even if you already found a number of records matching the limit but it will find only the first rows up to limit but it can possibly scan the entire table if there are not enough to fill up the limit

Unsmart•8/18/23, 5:49 PM

But like if you have 100 rows with 50% being value "1" and 50% being "2" and you do a where value = "1" if you get insanely lucky with the scan it could read as low as 50 rows and very bad luck would read all 100. something like that

UUnsmart Yeah there needs to be some order but I think we are both thinking the same thin...

Matt SilverlockOP•8/18/23, 5:49 PM

yep, exactly. It could scan the whole table and only find 5 results for

LIMIT 10

LIMIT 10

UUnsmart But like if you have 100 rows with 50% being value "1" and 50% being "2" and you...

Matt SilverlockOP•8/18/23, 5:51 PM

Yep! Just Binary Search Things

UUnsmart hmm that seems weird I feel like order shouldnt matter it would just scan and th...

Matt SilverlockOP•8/18/23, 5:54 PM

To add: you can have a sorted index (in D1 it doesn't matter which direction; the query engine can just scan the other way) which will further improve index efficiency if you are doing a lot of

ORDER BY <some_date_column>

ORDER BY <some_date_column>

queries. I probably need to add this to the docs.

MMatt Silverlock To add: you _can_ have a sorted index (in D1 it doesn't matter which direction; ...

Matt SilverlockOP•8/18/23, 5:55 PM

some reading if you're interested: https://stackoverflow.com/a/57082926/556573

Stack Overflow

Sort order for sqlite index

There is sort order when creating index for sqlite.

https://sqlite.org/lang_createindex.html
Each column name or expression can be followed by one of the "ASC" or "DESC" keywords to indicate ...

Unsmart•8/18/23, 5:57 PM

yeah thanks

MMatt Silverlock e.g. ``` ➜ wrangler d1 execute db-wnam --command "SELECT * FROM [Order] WHERE S...

Hello, I’m Allie!•8/18/23, 6:39 PM

Thanks for clarifying!

chientrm.com•8/18/23, 8:12 PM

do we need more than

SQL LIKE

SQL LIKE

MMatt Silverlock D1 now returns `rows_written` and `rows_read` in the metadata for every query. D...

Matt SilverlockOP•8/18/23, 9:53 PM

https://developers.cloudflare.com/d1/changelog/#row-count-now-returned-per-query

Changelog · Cloudflare D1 docs

D1 now returns a count of rows_written and rows_read for every query executed, allowing you to assess the cost of query for both pricing and index …

PPaganMuffin If I understand it correctly. e.g. ```SELECT * FROM table WHERE index_column =...

Matt SilverlockOP•8/18/23, 10:54 PM

I don’t know where you get 51 from? Your second example doesn’t seem right either. The actual behavior depends on if the index is ordered as well.

MMatt Silverlock D1 now returns `rows_written` and `rows_read` in the metadata for every query. D...

TigersWay•8/18/23, 11:02 PM

I just had a quick check to read those new

rows_written

rows_written

and

rows_read

rows_read

and I am kinda surprised to find written rows in a SELECT query

TigersWay•8/18/23, 11:03 PM

And way more read than there is rows!

TTigersWay I just had a quick check to read those new `rows_written` and `rows_read` and I ...

Matt SilverlockOP•8/18/23, 11:23 PM

What is your query? A SELECT isn’t guaranteed to be not write rows.

Unsmart•8/18/23, 11:28 PM

sql is confusing

CCole Xemi Is this going to be visible in the dashboard eventually?

Matt SilverlockOP•8/19/23, 12:26 AM

Yes. Within the next week or two

Hello, I’m Allie!•8/19/23, 7:32 AM

D1 Pricing

Pricing · Cloudflare D1 docs

While in public Alpha, D1 is currently free to use on all Workers plans. Refer to the our recent announcement ) for more information.

jahir9991•8/19/23, 7:59 AM

In production pages my get is working but writhing on db is not working is says no authorization can anyone aware of that i am using Sveltekit and drizzle and all binding are okay as i can get the data , my local dev is also working fine

kian•8/19/23, 8:06 AM

Do you have an example of a query that is failing?

jahir9991•8/19/23, 8:26 AM

Put and post is failing

jahir9991•8/19/23, 8:26 AM

It says not authorized

Kkian Do you have an example of a query that is failing?

jahir9991•8/19/23, 8:28 AM

Do i need to set any env variable rather than d1 binding

Kkian Do you have an example of a query that is failing?

jahir9991•8/19/23, 8:30 AM

https://staging.jahir.pages.dev/api/skills?q= this is get request which is working fine and serving data from d1

TigersWay•8/19/23, 8:36 AM

Following the apparition of

rows_written

rows_written

and

rows_read

rows_read

in every result, could we imagine some

total_rows_written

total_rows_written

and

total_rows_read

total_rows_read

in a "status" request?

Original message was deleted

Matt SilverlockOP•8/19/23, 8:38 AM

Keep in mind D1 won’t charge extra for replication (and it won’t multiply storage) or Time Travel (point in time recovery). $5/mo gets you 50M writes and 25B reads, 30 days Time Travel and storage.

TTigersWay Following the apparition of `rows_written` and `rows_read` in every result, coul...

Matt SilverlockOP•8/19/23, 8:39 AM

Can you expand? All of this will be in the GraphQL analytics API & dashboard

TTigersWay Following the apparition of `rows_written` and `rows_read` in every result, coul...

Matt SilverlockOP•8/19/23, 8:39 AM

e.g. https://developers.cloudflare.com/d1/platform/metrics-analytics/

Metrics and analytics · Cloudflare D1 docs

D1 exposes database analytics that allow you to inspect query volume, response size, and query latency across all and/or each database in your …

TigersWay•8/19/23, 8:40 AM

I guess I'll have to jump in that one! Thanks

Kkian Do you have an example of a query that is failing?

jahir9991•8/19/23, 11:20 AM

a hello @kiannh thanks for your replay , i got the issue , i added transaction in my query !!!!!!!!!!!!!, and its become rude ,
any way to use transaction ?

chientrm.com•8/19/23, 12:34 PM

Consider D1 is so superior, what's holding cloudflare

back from pushing the limits over 500MB?

Jjahir9991 a hello @kiannh thanks for your replay , i got the issue , i added ***transact...

Cyb3r-Jak3•8/19/23, 1:37 PM

Not currently. It is planned but no ETA. If you use

.batch()

.batch()

it gets auto transaction

chientrm.com•8/19/23, 4:40 PM

I'm using

EXPLAIN

EXPLAIN

statement. The column name is messed up?

Screen_Shot_2023-08-19_at_11.39.44_PM.png

Cchientrm.com Consider D1 is so superior, what's holding <:cloudflare:1026272852900581496> bac...

DebianArch•8/19/23, 7:22 PM

Cloudflare-d1 is very promising, the only thing I'm concerned about as-well is the 500Mb limit

DebianArch•8/19/23, 7:23 PM

Cloudflare D1 is pretty beastly when paired with drizzle orm makes relational queries so easy

DDebianArch Cloudflare D1 is pretty beastly when paired with drizzle orm makes relational qu...

chientrm.com•8/20/23, 2:00 AM

Cchientrm.com I'm using `EXPLAIN` statement. The column name is messed up? 🤔

Matt SilverlockOP•8/20/23, 11:27 AM

That’s correct.

When the EXPLAIN keyword appears by itself it causes the statement to behave as a query that returns the sequence of virtual machine instructions it would have used to execute the command had the EXPLAIN keyword not been present. When the EXPLAIN QUERY PLAN phrase appears, the statement returns high-level information regarding the query plan that would have been used.

https://www.sqlite.org/lang_explain.html

MMatt Silverlock That’s correct. > When the EXPLAIN keyword appears by itself it causes the state...

chientrm.com•8/20/23, 11:30 AM

totally not understand

JJorgeDev Will this be increased in the future? Does anyone know?

Hello, I’m Allie!•8/20/23, 12:40 PM

Many of these limits will increase during D1’s public alpha. Join the #d1-database channel in the Cloudflare Developer Discord to keep up to date with changes.
From the Docs

Limits · Cloudflare D1 docs

Many of these limits will increase during D1’s public alpha. Join the #d1-open-alpha channel in the Cloudflare Developer Discord to keep up to date …

Has nothing to do with memory at all (that's orthogonal). For example: - You have an index on a `cre

Similar Threads