Guys my d1 `f19f4e90-2514-4f2f-9d09-13ece5212673` is down. I really need it back for my presentation

Original message was deleted

S

Sébastien Morel•2/28/24, 8:04 PM

that's something I don't know how to check, where is the DB ?

S

Sébastien Morel•2/28/24, 8:05 PM

here is an example of meta @nora

"result": {
"success": true,
"meta": {
"served_by": "v3-prod",
"duration": 0.1743,
"changes": 0,
"last_row_id": 0,
"changed_db": false,
"size_after": 25100288,
"rows_read": 1804,
"rows_written": 0
},
"results": [
{
"COUNT(*)": 1804
}
]

"result": {
"success": true,
"meta": {
"served_by": "v3-prod",
"duration": 0.1743,
"changes": 0,
"last_row_id": 0,
"changed_db": false,
"size_after": 25100288,
"rows_read": 1804,
"rows_written": 0
},
"results": [
{
"COUNT(*)": 1804
}
]

"result": {
"success": true,
"meta": {
"served_by": "v3-prod",
"duration": 0.1743,
"changes": 0,
"last_row_id": 0,
"changed_db": false,
"size_after": 25100288,
"rows_read": 1804,
"rows_written": 0
},
"results": [
{
"COUNT(*)": 1804
}
]

"result": {
"success": true,
"meta": {
"served_by": "v3-prod",
"duration": 0.1743,
"changes": 0,
"last_row_id": 0,
"changed_db": false,
"size_after": 25100288,
"rows_read": 1804,
"rows_written": 0
},
"results": [
{
"COUNT(*)": 1804
}
]

S

Sébastien Morel•2/28/24, 8:05 PM

383ms to get it

S

Sébastien Morel•2/28/24, 8:09 PM

wrangler d1 info

S

Sébastien Morel•2/28/24, 8:09 PM

got it so same region with my worker checking the header CF-Placement

S

Sébastien Morel•2/28/24, 8:11 PM

also for SELECT query is it not supposed to be distributed?

AAaryan Guys my d1 `f19f4e90-2514-4f2f-9d09-13ece5212673` is down. I really need it back...

A

AaryanOP•2/28/24, 8:22 PM

seems to be back

S

Sébastien Morel•2/28/24, 8:29 PM

I am debugging a bit more and it seems to be the

_QUEUE?.send(payload)

_QUEUE?.send(payload)

_QUEUE?.send(payload)

_QUEUE?.send(payload) done right after the query that changes the game based on user location. Do queues have locations too?

W

wesbos•2/28/24, 8:31 PM

Hello - trying upload a DB. I followed the docs of dumping it

sqlite3 mydb.sqlite .dump > db.sql

sqlite3 mydb.sqlite .dump > db.sql

sqlite3 mydb.sqlite .dump > db.sql

sqlite3 mydb.sqlite .dump > db.sql and then import with

wrangler d1 execute podcast-stats-prod --file=db.sql

wrangler d1 execute podcast-stats-prod --file=db.sql

wrangler d1 execute podcast-stats-prod --file=db.sql

wrangler d1 execute podcast-stats-prod --file=db.sql but I get

not authorized [code: 7500]

not authorized [code: 7500]

not authorized [code: 7500]

not authorized [code: 7500] error.

Seems to be related to this? https://github.com/cloudflare/workers-sdk/issues/3892 Any ideas how to work around it? Its a 10,000 line long file so Im notsure where to look

GitHub

🐛 BUG: d1 execution with comment including semicolon causes SQL cod...

Which Cloudflare product(s) does this pertain to? D1 What version(s) of the tool(s) are you using? 3.6.0 What version of Node are you using? 20.5.1 What operating system are you using? Mac Describe...

Wwesbos Hello - trying upload a DB. I followed the docs of dumping it `sqlite3 mydb.sqli...

M

Max (@rozenmd)•2/28/24, 8:33 PM

Hey Wes,

not authorized

not authorized

gets thrown by the runtime when an sql command we haven't allow-listed gets used - look for things like

pragma

pragma

pragma

pragma statements

W

wesbos•2/28/24, 8:34 PM

Ah,

PRAGMA foreign_keys=OFF;

PRAGMA foreign_keys=OFF;

PRAGMA foreign_keys=OFF;

PRAGMA foreign_keys=OFF; is the first line

W

wesbos•2/28/24, 8:34 PM

ill remove it

W

wesbos•2/28/24, 8:35 PM

Hmm no dice - still errors

W

wesbos•2/28/24, 8:36 PM

could it be this at the start too?

BEGIN TRANSACTION;
CREATE TABLE _cf_KV (
      key TEXT PRIMARY KEY,
      value BLOB
    ) WITHOUT ROWID;

BEGIN TRANSACTION;
CREATE TABLE _cf_KV (
      key TEXT PRIMARY KEY,
      value BLOB
    ) WITHOUT ROWID;

BEGIN TRANSACTION;
CREATE TABLE _cf_KV (
      key TEXT PRIMARY KEY,
      value BLOB
    ) WITHOUT ROWID;

BEGIN TRANSACTION;
CREATE TABLE _cf_KV (
      key TEXT PRIMARY KEY,
      value BLOB
    ) WITHOUT ROWID;

Wwesbos could it be this at the start too? ```sql BEGIN TRANSACTION; CREATE TABLE _cf_K...

M

Max (@rozenmd)•2/28/24, 8:37 PM

Ah yep - remove the

BEGIN TRANSACTION;

BEGIN TRANSACTION;

BEGIN TRANSACTION;

BEGIN TRANSACTION; and the end of it, we're already running in a transaction for you (i'll also update the docs to mention that)

W

wesbos•2/28/24, 8:38 PM

Maybe im doing this wrong - I have a local wrangler D1 DB, I ran that dump command on the sqlite DB directly, but is there a wrangler command to push my local DB up to production? Or a command to export

M

Max (@rozenmd)•2/28/24, 8:39 PM

That is a good point though - wonder if we can't make wrangler take the local db and do that for you

W

wesbos•2/28/24, 8:45 PM

took the transaction stuff out and still hitting it. Will dig and se if there are any other lines

Original message was deleted

W

wesbos•2/28/24, 9:03 PM

No - should I have?

M

Max (@rozenmd)•2/28/24, 9:05 PM

and

COMMIT;

COMMIT;

COMMIT;

COMMIT;

M

Max (@rozenmd)•2/28/24, 9:06 PM

we'll add a note to replace these here in the meanwhile https://developers.cloudflare.com/d1/build-databases/import-data/#convert-sqlite-database-files

W

wesbos•2/28/24, 9:21 PM

it worked! Thank you

MMax (@rozenmd)we'll add a note to replace these here in the meanwhile https://developers.cloud...

I

Isaac McFadyen•2/28/24, 10:26 PM

The COMMIT part is under https://developers.cloudflare.com/d1/build-databases/import-data/#troubleshooting but yeah, good to move up a section too

M

matt•2/28/24, 11:27 PM

can you query d1 from outside a cloudflare worker?

M

matt•2/29/24, 12:05 AM

thanks @geelen for my use case I have a kafka topic that I want to consume from & then write results to d1. I guess my best option is to run a cloudflare worker on a cron trigger every 1 minute to consume, process & write to d1

M

matt•2/29/24, 12:18 AM

so this is coming off the back of a realtime streams system (bytewax) so I guess I would create a new flow that pulls from the kafka topic and then writes to d1

M

matt•2/29/24, 12:19 AM

and then deploy it onto my k8s cluster. im building a zapier-like system so d1 would be one of the potential places where data is exported

M

matt•2/29/24, 12:21 AM

I think I might do that, then it'll also reduce the coupling between the service & the cloudflare environment as it'll just need to know url + api key

M

matt•2/29/24, 12:22 AM

i think i might avoid cron as there's a chance in future i need to work on making this ultra-low-latency

M

matt•2/29/24, 12:23 AM

thanks for the advice

HHyperLapse Hi! I want to ask little but important question for me :) According https://deve...

M

Max (@rozenmd)•2/29/24, 12:42 PM

D1 is built to work to the strengths of Cloudflare's network, so we can't offer extremely large databases while also loading them quickly. It's significantly easier for us to offer millions of DBs with a size limit instead.

The limit is currently 2GB and we're working towards making it 10GB - but D1 will require a change in how you think about your project's data if you're looking to build a big platform. Instead of storing all of your data in a single DB, you would store each customer's data in their own DB (and we're building tooling to make it easier to dynamically access multiple DBs in a single Worker without redeploying)

MMax (@rozenmd)D1 is built to work to the strengths of Cloudflare's network, so we can't offer ...

J

João Castro•2/29/24, 2:50 PM

Are you also building tooling to help with executing migrations on a group of databases?

JJoão Castro Are you also building tooling to help with executing migrations on a group of da...

M

Max (@rozenmd)•2/29/24, 2:50 PM

as a follow-up to dynamic bindings, yes

I

itsmatteomanf•2/29/24, 3:07 PM

I'm getting

"error": "D1_ERROR: D1 is overloaded. Requests queued for too long."

"error": "D1_ERROR: D1 is overloaded. Requests queued for too long."

"error": "D1_ERROR: D1 is overloaded. Requests queued for too long."

"error": "D1_ERROR: D1 is overloaded. Requests queued for too long." on delete queries, some do succeed, some don't but I technically don't know unless I check row counts...

I

itsmatteomanf•2/29/24, 3:08 PM

It's a lot of rows (100s of thousands), but with the only filtering clause on a column with an index...

Iitsmatteomanf I'm getting `"error": "D1_ERROR: D1 is overloaded. Requests queued for too long....

M

Max (@rozenmd)•2/29/24, 3:34 PM

whats the db id again?

MMax (@rozenmd)whats the db id again?

I

itsmatteomanf•2/29/24, 3:35 PM

546787dd-35d5-4822-9b50-36fa0a4310c2

546787dd-35d5-4822-9b50-36fa0a4310c2

546787dd-35d5-4822-9b50-36fa0a4310c2

546787dd-35d5-4822-9b50-36fa0a4310c2

Iitsmatteomanf `546787dd-35d5-4822-9b50-36fa0a4310c2`

M

Max (@rozenmd)•2/29/24, 3:40 PM

were you doing a delete on top of the regular task that happens every hour?

MMax (@rozenmd)were you doing a delete on top of the regular task that happens every hour?

I

itsmatteomanf•2/29/24, 3:41 PM

Yes, I was... might have been just after it started, to be fair, I didn't check the time.

M

Max (@rozenmd)•2/29/24, 3:41 PM

looks like thats what tipped it into overloading

Guys my d1 `f19f4e90-2514-4f2f-9d09-13ece5212673` is down. I really need it back for my presentation

Similar Threads

Similar Threads

Similar Threads