@elithrar Is it expected to take more than an hour to process ~31K INSERT INTO statements from the .

MMozzy @elithrar Is it expected to take more than an hour to process ~31K INSERT INTO s...

M

It'll be

wrangler

wrangler

trying to parse your file. We haven't optimized this yet (it's unrelated to D1 / the storage backend). Break it up into ~5k line files.

MMozzy @elithrar Is it expected to take more than an hour to process ~31K INSERT INTO s...

M

Max (@rozenmd)•5/19/23, 3:11 PM

try pasting it into the console - the ui will lag, but insert should go through

MMatt Silverlock It'll be `wrangler` trying to parse your file. We haven't optimized this yet (it...

M

Matt Silverlock•5/19/23, 3:11 PM

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

~/repos/d1-demo (master) ✗ 
➜  npx wrangler@3 d1 execute key-value-db --file src/kv-5k.sql           

🌀 Mapping SQL input into an array of statements
🌀 Parsing 5000 statements
🌀 Executing on key-value-db (a197cf87-47ad-4473-b900-4c59c42f509d):
🚣 Executed 5000 commands in 201.99864504113793ms

M

MozzyOP•5/19/23, 3:11 PM

Thanks, I'll try both

M

MozzyOP•5/19/23, 3:29 PM

Splitting the file has led to success

S

SmallShen•5/19/23, 3:58 PM

I want to know the performance difference between json query and native query, are there any benchmarks?

S

SmallShen•5/19/23, 4:00 PM

It looks great to make d1 a “schema-less” database, but performance is important

M

MozzyOP•5/19/23, 4:02 PM

Sadly seeing very random

Error: D1_ERROR

Error: D1_ERROR

Error: D1_ERROR

Error: D1_ERRORs
Sometimes it responds within 150ms, other times it errors

M

MozzyOP•5/19/23, 4:04 PM

Did anything regarding the non-experimental D1 change today as well?
This is the file it's failing in: https://github.com/leonlarsson/bfstats-api/blob/main/handlers/cacheD1.ts

M

MozzyOP•5/19/23, 4:05 PM

Specifically started having issues around this time

MMozzy Sadly seeing very random `Error: D1_ERROR`s Sometimes it responds within 150ms, ...

M

Matt Silverlock•5/19/23, 4:12 PM

Nothing regarding the original alpha changed (at all).

MMozzy Did anything regarding the non-experimental D1 change today as well? This is the...

M

Matt Silverlock•5/19/23, 4:13 PM

If your database has grown, then it's likely you're just running into the limitations of the original alpha

SSmallShen It looks great to make d1 a “schema-less” database, but performance is important

M

Matt Silverlock•5/19/23, 4:14 PM

It's going to really depend on what you're querying and what the shape of the objects you're using is. You also need to take into account the benefits of doing this in your database vs. in your application.

Ssourabh.sashank "D1Cause-GetAllGuruData","value":"Error: Network connection lost. after moving D...

M

Matt Silverlock•5/19/23, 4:19 PM

Can you share more information please? Specifically: your database ID, the query itself (with any PII redacted), and whether this is repeatable?

S

Sameer Ali•5/19/23, 4:22 PM

@elithrar just read the blog on D1 updates, great job guys, its looking promising. I have been waiting to hear about how we can create database per customer, however, nothing was mentioned about that in the blog. are you guys still considering that in near future or that will be available after GA?

MMatt Silverlock If your database has grown, then it's likely you're just running into the limita...

M

MozzyOP•5/19/23, 4:22 PM

Hmm, it doesn't grow super quickly. Maybe a huge coincidence then

M

MozzyOP•5/19/23, 4:23 PM

The entire DB is about 9mb

SSameer Ali @elithrar just read the blog on D1 updates, great job guys, its looking promisin...

M

Matt Silverlock•5/19/23, 4:35 PM

We'll be increasing limits (on DB size and total # of DBs) over the coming months. Are you using Workers for Platforms?

Ssourabh.sashank **Database ID:** c50f771b-40de-4432-a052-b1ffd73b5af3 **Query:** SELECT * FROM ...

M

Matt Silverlock•5/19/23, 4:35 PM

Thanks. Looking into it. May not be a quick fix.

MMatt Silverlock We'll be increasing limits (on DB size and total # of DBs) over the coming month...

D

DaniFoldi•5/19/23, 4:38 PM

I might be

MMozzy Did anything regarding the non-experimental D1 change today as well? This is the...

M

MozzyOP•5/19/23, 5:27 PM

D1 Beta doesn't like it either Sadge

M

MozzyOP•5/19/23, 5:33 PM

Error: Error: Network connection lost.
    at D1Database._send (index.js:3290:16)
    at async D1PreparedStatement.all (index.js:3355:7)
    at async cacheD1_default (index.js:3169:34)       
    at async Object.fetch (index.js:3185:7) {
  stack: Error: Error: Network connection lost.
    at D1Da…9:34)
    at async Object.fetch (index.js:3185:7),
  message: Error: Network connection lost.
}

Error: Error: Network connection lost.
    at D1Database._send (index.js:3290:16)
    at async D1PreparedStatement.all (index.js:3355:7)
    at async cacheD1_default (index.js:3169:34)       
    at async Object.fetch (index.js:3185:7) {
  stack: Error: Error: Network connection lost.
    at D1Da…9:34)
    at async Object.fetch (index.js:3185:7),
  message: Error: Network connection lost.
}

Error: Error: Network connection lost.
    at D1Database._send (index.js:3290:16)
    at async D1PreparedStatement.all (index.js:3355:7)
    at async cacheD1_default (index.js:3169:34)       
    at async Object.fetch (index.js:3185:7) {
  stack: Error: Error: Network connection lost.
    at D1Da…9:34)
    at async Object.fetch (index.js:3185:7),
  message: Error: Network connection lost.
}

Error: Error: Network connection lost.
    at D1Database._send (index.js:3290:16)
    at async D1PreparedStatement.all (index.js:3355:7)
    at async cacheD1_default (index.js:3169:34)       
    at async Object.fetch (index.js:3185:7) {
  stack: Error: Error: Network connection lost.
    at D1Da…9:34)
    at async Object.fetch (index.js:3185:7),
  message: Error: Network connection lost.
}

M

MozzyOP•5/19/23, 5:36 PM

Coincidence or not, I just got the same error in a completely different place: https://github.com/leonlarsson/bfstats-api/blob/main/handlers/D1Handler.ts#L58

J

Jason Hostetter•5/19/23, 5:58 PM

Trying to load my alpha db dump into the new backend via wrangler, with 27k or so lines, first just hung as it did for @mozzy , tried breaking into a file with 4999 lines, now getting

[ERROR] A request to the Cloudflare API (/accounts/c6ad2463148cc51ec38821637c5af28f/d1/database/f90e9b86-b9df-4257-b400-827dac122d25/query) failed.

  statement too long [code: 7500]

[ERROR] A request to the Cloudflare API (/accounts/c6ad2463148cc51ec38821637c5af28f/d1/database/f90e9b86-b9df-4257-b400-827dac122d25/query) failed.

  statement too long [code: 7500]

[ERROR] A request to the Cloudflare API (/accounts/c6ad2463148cc51ec38821637c5af28f/d1/database/f90e9b86-b9df-4257-b400-827dac122d25/query) failed.

  statement too long [code: 7500]

[ERROR] A request to the Cloudflare API (/accounts/c6ad2463148cc51ec38821637c5af28f/d1/database/f90e9b86-b9df-4257-b400-827dac122d25/query) failed.

  statement too long [code: 7500]

J

Jason Hostetter•5/19/23, 6:00 PM

Tried pasting entire 27k sql lines into the web console, after a minute or two all the statements showed up in the console but the last line is again:

statement too long

statement too long

statement too long

statement too long

Original message was deleted

N

nickchomey•5/19/23, 6:12 PM

I'm confused about the db size limits. The blog says in the pricing that 1GB is available to free users and 5GB included and $0.75/GB beyond that.

But then it says "Developerson the Workers Paid plan will get access to 1GB dbs in the very near future".

Does this mean that the limit is currently still 100mb, will soon be 1Gb and later will change to what the pricing shows?

Ffidozo Am I crazy to ask for one D1 database per user on my app? Similar to DOs, the D1...

B

Bulbasaur•5/19/23, 6:16 PM

maybe a better approach would be dbs per regional area

B

Bulbasaur•5/19/23, 6:17 PM

when a user creates their account, pick a db based on their geography to route to, and always use that db for that user

B

Bulbasaur•5/19/23, 6:18 PM

depending on how far you want to take that, you could do an NA region, EU region and Asia region for example, or more

B

Bulbasaur•5/19/23, 6:19 PM

Location Hints will allow you to peg where the write region is placed

Nnickchomey I'm confused about the db size limits. The blog says in the pricing that 1GB is ...

I

Isaac McFadyen•5/19/23, 6:28 PM

Yes. Limit is still 100mb, will be raised to 1GB, and then likely further eventually.

Ttouhidurrr 🇵🇸i want to access d1 not from workers but from api, is that possible? if yes then...

I

Isaac McFadyen•5/19/23, 6:28 PM

Yes it's possible but undocumented as far as I know (and the API is slower than D1 from a Worker)

Nnickchomey I'm confused about the db size limits. The blog says in the pricing that 1GB is ...

U

Unsmart•5/19/23, 6:28 PM

Currently in alpha it is free to use and limited to 100mb. In the future those payment plans will come into play and the limits for dbs will be higher

I

Isaac McFadyen•5/19/23, 6:28 PM

You can look at how I'm doing it here: https://github.com/isaac-mcfadyen/d1-console

I

Isaac McFadyen•5/19/23, 6:28 PM

(think those endpoints should still work)

UUnsmart Currently in alpha it is free to use and limited to 100mb. In the future those p...

N

nickchomey•5/19/23, 6:33 PM

Thanks! Very exciting stuff

DDong Another use case is that I have multiple applications needs to fetch data from t...

M

Matt Silverlock•5/19/23, 7:26 PM

Yep, you can bind to a D1 DB from multiple Workers, just like Queues/KV/R2/etc.

JJason Hostetter Trying to load my alpha db dump into the new backend via wrangler, with 27k or s...

J

Jason Hostetter•5/19/23, 7:46 PM

Looks like it was one of the statements in my dumped/converted .sql file that was inserting a very long binary blob and this was throwing the error. If I skipped that insert statement, the rest of it imported alright

M

Murder Chicken•5/19/23, 8:04 PM

Any ideas what might be causing this or a way to debug this issue a bit more? https://github.com/cloudflare/workers-sdk/issues/3294

GitHub

🐛 BUG: wrangler pages dev does not work with D1 · Issue #3294 · clo...

Which Cloudflare product(s) does this pertain to? Wrangler What version of Wrangler are you using? 3.0.0 What operating system are you using? Windows Describe the Bug create a fresh D1 local DB wit...

JJason Hostetter Looks like it was one of the statements in my dumped/converted .sql file that wa...

M

Matt Silverlock•5/19/23, 8:05 PM

How long? https://developers.cloudflare.com/d1/platform/limits/

I bet you hit the max row length with a big blob. Would be helpful to know so we can think about adjusting over time.

MMatt Silverlock How long? https://developers.cloudflare.com/d1/platform/limits/ - I bet you hit ...

J

Jason Hostetter•5/19/23, 8:14 PM

267k characters

@elithrar Is it expected to take more than an hour to process ~31K INSERT INTO statements from the .

Similar Threads

Similar Threads

Similar Threads