Is there a limit on the size of the database we can export ? I've been trying to run the command on

SSoftbear Is there a limit on the size of the database we can export ? I've been trying to...

S

SoftbearOP•5/7/24, 3:39 PM

The error on export was fixed by updating wrangler to the latest version.

But now I have a 504 error on import.
There is around 28k rows in a table so it breaks it down in 3 batches, the 3rd one does not go trough (around 18k rows are imported).

SSoftbear The error on export was fixed by updating wrangler to the latest version. But ...

R

roftt•5/7/24, 3:58 PM

The same happened to me and my database is 5 MB only. Does your rows contain lots of text? Most likely this bug: https://github.com/cloudflare/workers-sdk/issues/4510

S

SoftbearOP•5/7/24, 4:00 PM

No pretty small rows

INSERT INTO tokens VALUES('XXX',2024-01-27 08:56:38,2024-01-27 08:56:38,'XXX','XXX','ACCESS',NULL,NULL);

INSERT INTO tokens VALUES('XXX',2024-01-27 08:56:38,2024-01-27 08:56:38,'XXX','XXX','ACCESS',NULL,NULL);

INSERT INTO tokens VALUES('XXX',2024-01-27 08:56:38,2024-01-27 08:56:38,'XXX','XXX','ACCESS',NULL,NULL);

INSERT INTO tokens VALUES('XXX',2024-01-27 08:56:38,2024-01-27 08:56:38,'XXX','XXX','ACCESS',NULL,NULL);

SSoftbear No pretty small rows `INSERT INTO tokens VALUES('XXX',2024-01-27 08:56:38,2024...

R

roftt•5/7/24, 4:02 PM

Have no idea then. Ended up importing each table separately

S

Sam•5/7/24, 7:00 PM

Hey all, is something up with

wrangler d1 info

wrangler d1 info

wrangler d1 info

wrangler d1 info +

wrangler d1 insights

wrangler d1 insights

wrangler d1 insights

wrangler d1 insights?

info

info

info

info is reporting 0 read queries in the past 24h (not the case unless it's somehow using some local version? The ID is the same as in my CF dash)

insights

insights

insights

insights is returning an empty array regardless of configuration

S

SoftbearOP•5/7/24, 8:10 PM

I'm getting timeout errors when trying to modify the schema of a table. " Error: D1_ERROR: D1 exceeded its CPU time limit and was reset."

The users table has 18k rows and 6 tables have foreign key references on it so that's probably why its taking a lot of time. But because of this I'm kind of stuck with not being able to modify my users table.

D

di_uthred•5/8/24, 12:28 AM

Having an issue with this D1 insert. The DB binding is fine (another function reads from the DB OK), but this insert just will not execute. It gets to the "Preparing to insert.." line OK and then black hole - not even an exception error. Simplifying by stubbing in values to avoid type mismatch issues didn't help either.

const insertSql =


        INSERT INTO company_profile 
        (symbol, companyName, website, fullTimeEmployees, industry, overallRisk, description) 
        VALUES (?, ?, ?, ?, ?, ?, ?);


        INSERT INTO company_profile 
        (symbol, companyName, website, fullTimeEmployees, industry, overallRisk, description) 
        VALUES (?, ?, ?, ?, ?, ?, ?);


        INSERT INTO company_profile 
        (symbol, companyName, website, fullTimeEmployees, industry, overallRisk, description) 
        VALUES (?, ?, ?, ?, ?, ?, ?);


        INSERT INTO company_profile 
        (symbol, companyName, website, fullTimeEmployees, industry, overallRisk, description) 
        VALUES (?, ?, ?, ?, ?, ?, ?);

;

for (const entry of data) {
console.log(

Preparing to insert/update data for symbol: ${entry.symbol}

Preparing to insert/update data for symbol: ${entry.symbol}

Preparing to insert/update data for symbol: ${entry.symbol}

Preparing to insert/update data for symbol: ${entry.symbol});
try {
const stmt = await DB.prepare(insertSql);
await stmt.bind(entry.symbol, entry.companyName, entry.website, entry.fullTimeEmployees, entry.industry, entry.overallRisk, entry.description);
const result = await stmt.run();

if (result) {
console.log(

Data successfully inserted for symbol: ${entry.symbol}

Data successfully inserted for symbol: ${entry.symbol}

Data successfully inserted for symbol: ${entry.symbol}

Data successfully inserted for symbol: ${entry.symbol}, result);
} else {
console.log(

Insert failed for symbol: ${entry.symbol}

Insert failed for symbol: ${entry.symbol}

Insert failed for symbol: ${entry.symbol}

Insert failed for symbol: ${entry.symbol}, result);
}
} catch (error) {
console.error(

Failed to insert data for symbol: ${entry.symbol}

Failed to insert data for symbol: ${entry.symbol}

Failed to insert data for symbol: ${entry.symbol}

Failed to insert data for symbol: ${entry.symbol}, error);
}
}

Ddi_uthred Having an issue with this D1 insert. The DB binding is fine (another function r...

R

roftt•5/8/24, 6:24 AM

should be VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)

SSan possible bug: I noticed that when an account member has "Administrator Read Only...

U

Utkarsh Saxena•5/8/24, 7:08 AM

Any update on this?

N

Natalie•5/8/24, 9:51 AM

Hi there. I'm building an realtime 3d texturing application and am planning on utilising cloudflare D1 as the backing store. My problem is that given the limit of 50k databases each with a max size of 10GB, I'm unsure what would be the best architecture to adopt. My initial instinct would be to create a DB per organization, but that means I could only have up to 50k orgs. Given a pareto assumption of organization, probably most of these databases would be under-utilized. So an alternative would be to shard the data somehow, but that could quickly becomes awkward if remote joins would are needed. So I'm a bit confused about how D1 fits into the SaaS use case

N

Natalie•5/8/24, 9:54 AM

I would also be using durable objects as well. But for non-realtime data, would still need a traditional database

NNatalie Hi there. I'm building an realtime 3d texturing application and am planning on u...

H

Hello, I’m Allie!•5/8/24, 9:55 AM

IIRC the 50k DB limit should be able to be raised on request

N

Natalie•5/8/24, 9:58 AM

That also begs the question of if it is possible to have dynamic db bindings? As 50k DBs would far exceed the max number of bindings per worker

NNatalie That also begs the question of if it is possible to have dynamic db bindings? As...

H

Hello, I’m Allie!•5/8/24, 10:00 AM

Not at the moment, though iirc it is something the team is working on

N

Natalie•5/8/24, 10:01 AM

Would that use the rest API?

H

Hello, I’m Allie!•5/8/24, 10:01 AM

It may be ok with WfP? IIRC you should be able to bind to an individual script

NNatalie Would that use the rest API?

H

Hello, I’m Allie!•5/8/24, 10:04 AM

No, I believe it would be like a Durable Object, where you pass an ID in, and get a binding out

H

Hello, I’m Allie!•5/8/24, 10:05 AM

Though I'm not on the team, so that is just a guess on my part

N

Natalie•5/8/24, 10:05 AM

Ah. Makes sense

N

Natalie•5/8/24, 10:05 AM

Thanks

K

krofdrakula•5/8/24, 1:38 PM

does anyone know if prepared statements can be bound with arrays? i'm trying to do a query like

db.prepare('SELECT * FROM table WHERE status IN (?)').bind(['active','expired']).all()

db.prepare('SELECT * FROM table WHERE status IN (?)').bind(['active','expired']).all()

, would that work?

Kkrofdrakula does anyone know if prepared statements can be bound with arrays? i'm trying to ...

R

roftt•5/8/24, 1:41 PM

https://discord.com/channels/595317990191398933/992060581832032316/1226795894280687636

M

Marcel Overdijk•5/8/24, 3:06 PM

The pricing docs for D1 mention:

Every query returns a meta object that contains a total count of the rows read (rows_read) and rows written (rows_written) by that query. For example, a query that performs a full table scan (for instance, SELECT * FROM users) from a table with 5000 rows would return a rows_read value of 5000:

"meta": {
  "duration": 0.20472300052642825,
  "size_after": 45137920,
  "rows_read": 5000,
  "rows_written": 0
}

These are also included in the D1 Cloudflare dashboard and the analytics API, allowing you to attribute read and write volumes to specific databases, time periods, or both.

Every query returns a meta object that contains a total count of the rows read (rows_read) and rows written (rows_written) by that query. For example, a query that performs a full table scan (for instance, SELECT * FROM users) from a table with 5000 rows would return a rows_read value of 5000:

"meta": {
  "duration": 0.20472300052642825,
  "size_after": 45137920,
  "rows_read": 5000,
  "rows_written": 0
}

These are also included in the D1 Cloudflare dashboard and the analytics API, allowing you to attribute read and write volumes to specific databases, time periods, or both.

Every query returns a meta object that contains a total count of the rows read (rows_read) and rows written (rows_written) by that query. For example, a query that performs a full table scan (for instance, SELECT * FROM users) from a table with 5000 rows would return a rows_read value of 5000:

"meta": {
  "duration": 0.20472300052642825,
  "size_after": 45137920,
  "rows_read": 5000,
  "rows_written": 0
}

These are also included in the D1 Cloudflare dashboard and the analytics API, allowing you to attribute read and write volumes to specific databases, time periods, or both.

Every query returns a meta object that contains a total count of the rows read (rows_read) and rows written (rows_written) by that query. For example, a query that performs a full table scan (for instance, SELECT * FROM users) from a table with 5000 rows would return a rows_read value of 5000:

"meta": {
  "duration": 0.20472300052642825,
  "size_after": 45137920,
  "rows_read": 5000,
  "rows_written": 0
}

These are also included in the D1 Cloudflare dashboard and the analytics API, allowing you to attribute read and write volumes to specific databases, time periods, or both.

But where can I see these rows_read statitiscs in the dashboard?

I'm trying to find out how count queries are calculated...

Imagine I have a

select count(1) from customer where country_id = 'uk'

select count(1) from customer where country_id = 'uk'

what is the cost for such a query?
Is it equals to the number of rows in the customer table?

R

roftt•5/8/24, 3:24 PM

Try this in your worker:

const stmt = db.prepare('select count(1) from customer where country_id = "uk"').all();
console.log(stmt.meta.rows_read)

const stmt = db.prepare('select count(1) from customer where country_id = "uk"').all();
console.log(stmt.meta.rows_read)

const stmt = db.prepare('select count(1) from customer where country_id = "uk"').all();
console.log(stmt.meta.rows_read)

const stmt = db.prepare('select count(1) from customer where country_id = "uk"').all();
console.log(stmt.meta.rows_read)

Overall statistics for rows_read of the entire database is in Dashboard -> Your DB -> Metrics
And don't forget to use indexes, it reduces rows_read drastically

M

Marcel Overdijk•5/8/24, 5:27 PM

I'm wondering if I can check these query statistic per query executed from the console?

K

kian•5/8/24, 5:31 PM

If you open DevTools, the

meta

meta

object will be in the HTTP response.

M

Marcel Overdijk•5/8/24, 5:38 PM

That was indeed useful.
So when a

customer

customer

table contains 900 records both queries

select * from customer

select * from customer

and

select count(1) from customer

select count(1) from customer

will have

"rows_read": 900,

"rows_read": 900,

M

Marcel Overdijk•5/8/24, 5:39 PM

so

count

count

queries are quite expensive in D1

Rroftt should be VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)

D

di_uthred•5/8/24, 6:37 PM

didn't make a difference

MMarcel Overdijk so `count` queries are quite expensive in D1

I

Isaac McFadyen•5/8/24, 7:41 PM

This is not really specific to D1 (but yeah that's accurate)

I

Isaac McFadyen•5/8/24, 7:41 PM

A lot of databases will read every row for "count" because they assume you want an exact count.

I

Isaac McFadyen•5/8/24, 7:42 PM

I believe there's some other databases (I think CockroachDB?) that will give you approximate counts without reading every row, but SQLite (and therefore D1) does not do this.

U

Unsmart•5/8/24, 10:11 PM

From how I am reading it the 1 per 1000 thing relates to indexes not documents. So you are always charged at minimum 1 read per document + something for indexes

U

Unsmart•5/8/24, 10:12 PM

Firestore also costs like 600x the price for reads so

M

Marcel Overdijk•5/8/24, 10:14 PM

I understand as it needs to actually read the rows.
Maybe this could be added to docs explicitly in the pricing section?

U

Unsmart•5/8/24, 10:18 PM

It is in the doc on the pricing page:

1. Rows read measure how many rows a query reads (scans), regardless of the size of each row. For example, if you have a table with 5000 rows and run a SELECT * FROM table as a full table scan, this would count as 5,000 rows read. A query that filters on an unindexed column may return fewer rows to your Worker, but is still required to read (scan) more rows to determine which subset to return.

M

Marcel Overdijk•5/8/24, 10:18 PM

Another thing I'm wondering now is the foolowing:
I'm implementing an api with a traditional pagination using

offset

offset

,

limit

limit

(so no cursors).
Responses also include the

totalCount

totalCount

Currently I'm executing 2 queries:
- get the actual records for the given page (e.g. with page size

this will result in 10 rows being queried.
- the count query (so when having 1000 rows in the D1 database this will result in 1000 rows read)
together this will result in 1010 rows being read
I could in fact just query all rows (1000!! reads) and then do the filtering (pagination) in the code....

UUnsmart It is in the doc on the pricing page: > 1. Rows read measure how many rows a que...

M

Marcel Overdijk•5/8/24, 10:19 PM

I mean explicitly stating that a count query will also perform these reads (so count resulting in value 1000, will mean 1000 rows read)

M

Marcel Overdijk•5/8/24, 10:21 PM

Above does not seem effective but will cost 10 reads less... I'm not concerned about those 10 more reads btw.
But maybe executing 1 query instead of 2 might even perform better....

M

Marcel Overdijk•5/8/24, 10:23 PM

The tables I'm quering this way will not contain more than 1000 rows on average.

MMarcel Overdijk Another thing I'm wondering now is the foolowing: I'm implementing an api with a...

I

Isaac McFadyen•5/8/24, 10:29 PM

I'd strongly recommend against the "query all the rows" approach.

I

Isaac McFadyen•5/8/24, 10:30 PM

Workers have memory limits so if you filter in the Worker you'll still run into issues where if the rows are very large and a script is being executed by 2 people then you might hit those limits

I

Isaac McFadyen•5/8/24, 10:30 PM

Plus you'll be billed more for processing time in the Worker itself (which is likely more than the extra 10 reads from D1)

M

Marcel Overdijk•5/8/24, 10:49 PM

Yes I’m also against it.

M

Marcel Overdijk•5/8/24, 10:51 PM

But I might instead cache the result of the count query, as it is quite static data.

M

Marcel Overdijk•5/8/24, 10:56 PM

Yes that could be an option.
But I also have other filter arguments besides offset limit…
Like country, category etc. In the api 0, 1 or more of the filters can be applied.

M

Marcel Overdijk•5/8/24, 11:02 PM

No 0, 1 or more column based filters + limit / offset.
Where normally the limit is 10 so 10 rows are returned. But also the total count based on the filters is needed. (To show navigation buttons)

M

Marcel Overdijk•5/8/24, 11:03 PM

So result is 10 rows (based on limit param) but count could be 300, 400 or like 1000

Is there a limit on the size of the database we can export ? I've been trying to run the command on

Similar Threads

Similar Threads

Similar Threads