so `count` queries are quite expensive in D1

Rroftt should be VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)

D

di_uthred•5/8/24, 6:37 PM

didn't make a difference

MMarcel Overdijk so `count` queries are quite expensive in D1

I

Isaac McFadyen•5/8/24, 7:41 PM

This is not really specific to D1 (but yeah that's accurate)

I

Isaac McFadyen•5/8/24, 7:41 PM

A lot of databases will read every row for "count" because they assume you want an exact count.

I

Isaac McFadyen•5/8/24, 7:42 PM

I believe there's some other databases (I think CockroachDB?) that will give you approximate counts without reading every row, but SQLite (and therefore D1) does not do this.

U

Unsmart•5/8/24, 10:11 PM

From how I am reading it the 1 per 1000 thing relates to indexes not documents. So you are always charged at minimum 1 read per document + something for indexes

U

Unsmart•5/8/24, 10:12 PM

Firestore also costs like 600x the price for reads so

M

Marcel OverdijkOP•5/8/24, 10:14 PM

I understand as it needs to actually read the rows.
Maybe this could be added to docs explicitly in the pricing section?

U

Unsmart•5/8/24, 10:18 PM

It is in the doc on the pricing page:

1. Rows read measure how many rows a query reads (scans), regardless of the size of each row. For example, if you have a table with 5000 rows and run a SELECT * FROM table as a full table scan, this would count as 5,000 rows read. A query that filters on an unindexed column may return fewer rows to your Worker, but is still required to read (scan) more rows to determine which subset to return.

M

Marcel OverdijkOP•5/8/24, 10:18 PM

Another thing I'm wondering now is the foolowing:
I'm implementing an api with a traditional pagination using

offset

offset

offset

offset,

limit

limit

limit

limit (so no cursors).
Responses also include the

totalCount

totalCount

totalCount

totalCount
Currently I'm executing 2 queries:
- get the actual records for the given page (e.g. with page size

this will result in 10 rows being queried.
- the count query (so when having 1000 rows in the D1 database this will result in 1000 rows read)
together this will result in 1010 rows being read
I could in fact just query all rows (1000!! reads) and then do the filtering (pagination) in the code....

UUnsmart It is in the doc on the pricing page: > 1. Rows read measure how many rows a que...

M

Marcel OverdijkOP•5/8/24, 10:19 PM

I mean explicitly stating that a count query will also perform these reads (so count resulting in value 1000, will mean 1000 rows read)

M

Marcel OverdijkOP•5/8/24, 10:21 PM

Above does not seem effective but will cost 10 reads less... I'm not concerned about those 10 more reads btw.
But maybe executing 1 query instead of 2 might even perform better....

M

Marcel OverdijkOP•5/8/24, 10:23 PM

The tables I'm quering this way will not contain more than 1000 rows on average.

MMarcel Overdijk Another thing I'm wondering now is the foolowing: I'm implementing an api with a...

I

Isaac McFadyen•5/8/24, 10:29 PM

I'd strongly recommend against the "query all the rows" approach.

I

Isaac McFadyen•5/8/24, 10:30 PM

Workers have memory limits so if you filter in the Worker you'll still run into issues where if the rows are very large and a script is being executed by 2 people then you might hit those limits

I

Isaac McFadyen•5/8/24, 10:30 PM

Plus you'll be billed more for processing time in the Worker itself (which is likely more than the extra 10 reads from D1)

M

Marcel OverdijkOP•5/8/24, 10:49 PM

Yes I’m also against it.

M

Marcel OverdijkOP•5/8/24, 10:51 PM

But I might instead cache the result of the count query, as it is quite static data.

M

Marcel OverdijkOP•5/8/24, 10:56 PM

Yes that could be an option.
But I also have other filter arguments besides offset limit…
Like country, category etc. In the api 0, 1 or more of the filters can be applied.

M

Marcel OverdijkOP•5/8/24, 11:02 PM

No 0, 1 or more column based filters + limit / offset.
Where normally the limit is 10 so 10 rows are returned. But also the total count based on the filters is needed. (To show navigation buttons)

M

Marcel OverdijkOP•5/8/24, 11:03 PM

So result is 10 rows (based on limit param) but count could be 300, 400 or like 1000

W

wiln•5/8/24, 11:26 PM

Hello, I am new to D1, I want to ask if it is possible to read and write to the database using the python code on my computers? It makes me feel like you can only do that using javascript and workers

I

Isaac McFadyen•5/9/24, 3:30 AM

TIL, that's cool

U

Unsmart•5/9/24, 3:48 AM

neat, never made a table without an index personally so would always work well

M

matt•5/9/24, 7:43 AM

im a little confused by some of the language in the d1 material I've been reading -- are read replicas available now?

Mmatt im a little confused by some of the language in the d1 material I've been readin...

H

Hello, I’m Allie!•5/9/24, 7:44 AM

They are not

M

matt•5/9/24, 7:46 AM

ah ok thanks for the update. the blog post from april makes it sounds like they are already running. did CF provide any timelines for when they might be available? really like the look of the session API and can't wait to try it out

Mmatt ah ok thanks for the update. the blog post from april makes it sounds like they ...

H

Hello, I’m Allie!•5/9/24, 7:46 AM

:soontm:

H

Hello, I’m Allie!•5/9/24, 7:46 AM

Best answer we can give you atm

M

matt•5/9/24, 7:46 AM

no ones perfect

Ggeelen if you're doing `select count(1) from my_table` then as long as `my_table` has a...

M

Marcel OverdijkOP•5/9/24, 8:26 AM

Thanks for joining the discussion @geelen , however the behavior you mention is not what I see:

I have a table with a primary key on the

id

id

field and that table also has various other index like on

name

name

.
But this is the results I get:

With select *

{
  "result": [
    {
      "results": [
        {
          "count(*)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2931,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(*)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2931,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(*)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2931,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(*)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2931,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

With select 1:

{
  "result": [
    {
      "results": [
        {
          "count(1)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2308,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(1)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2308,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(1)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2308,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(1)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.2308,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

M

Marcel OverdijkOP•5/9/24, 8:31 AM

With select id (on primary key):

{
  "result": [
    {
      "results": [
        {
          "count(id)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4641,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(id)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4641,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(id)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4641,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(id)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4641,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

On other indexed field:

{
  "result": [
    {
      "results": [
        {
          "count(name)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4976,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(name)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4976,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(name)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4976,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(name)": 900
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.4976,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 900,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

But as you can see the

rows_read

rows_read

rows_read

rows_read is always similar to the count result.

M

Marcel OverdijkOP•5/9/24, 8:35 AM

M

Marcel OverdijkOP•5/9/24, 8:35 AM

als when showing the table details it indicates the

id

id

column is a PK

M

Marcel OverdijkOP•5/9/24, 8:40 AM

Also

PRAGMA index_list('driver');

PRAGMA index_list('driver');

PRAGMA index_list('driver');

PRAGMA index_list('driver'); gives me all the indexes (inclduing the unique PK index):

M

Marcel OverdijkOP•5/9/24, 8:43 AM

I also tried:

select count(country_of_birth_country_id) from driver where country_of_birth_country_id = 'united-kingdom'

select count(country_of_birth_country_id) from driver where country_of_birth_country_id = 'united-kingdom'

select count(country_of_birth_country_id) from driver where country_of_birth_country_id = 'united-kingdom'

select count(country_of_birth_country_id) from driver where country_of_birth_country_id = 'united-kingdom'
(

country_of_birth_country_id

country_of_birth_country_id

country_of_birth_country_id

country_of_birth_country_id is an indexed column and I use in the

count

count

and in the

where

where

) but it gives:

{
  "result": [
    {
      "results": [
        {
          "count(country_of_birth_country_id)": 174
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.7666,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 175,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(country_of_birth_country_id)": 174
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.7666,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 175,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(country_of_birth_country_id)": 174
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.7666,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 175,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

{
  "result": [
    {
      "results": [
        {
          "count(country_of_birth_country_id)": 174
        }
      ],
      "success": true,
      "meta": {
        "served_by": "v3-prod",
        "duration": 0.7666,
        "changes": 0,
        "last_row_id": 0,
        "changed_db": false,
        "size_after": 12607488,
        "rows_read": 175,
        "rows_written": 0
      }
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}

Result from the count was

174

174 but still

175

175 rows read.

M

Marcel OverdijkOP•5/9/24, 8:45 AM

I assume 174 + 1 as it accesses the index? But anyway, I cannot seem to get count queries to have only 1 read...

R

roftt•5/9/24, 2:26 PM

Trying to bind 3000 databases to my worker in wrangler.toml but getting "504 Gateway Time-out" error. It works for ~1800 but for 3000 it's hitting the time limit? What can I do?

K

kian•5/9/24, 2:26 PM

Bind less databases is the only real answer

K

kian•5/9/24, 2:26 PM

It'll take longer with each binding until it times out

W

Walshy•5/9/24, 2:27 PM

maybe wfp would be better suited here?

W

Walshy•5/9/24, 2:27 PM

If you're making a db per customer you can make a worker per customer and have the binding per one

so `count` queries are quite expensive in D1

Similar Threads

Similar Threads

Similar Threads