+1 to deploying it, `wrangler dev`'s Workers !== real Workers

Original message was deleted

L

Luka•2/14/24, 3:57 PM

Yes

Original message was deleted

L

Luka•2/14/24, 3:58 PM

That's great. But there's this one table that has around 90k rows and I suspect it being a few hundred MBs

MMax (@rozenmd)+1 to deploying it, `wrangler dev`'s Workers !== real Workers

L

Luka•2/14/24, 3:59 PM

to remove any ambiguity, are you talking about

worker dev --remote

worker dev --remote

?

LLuka to remove any ambiguity, are you talking about `worker dev --remote`? 🙂

M

Max (@rozenmd)OP•2/14/24, 3:59 PM

yes

MMax (@rozenmd)yes

L

Luka•2/14/24, 3:59 PM

so you're suggesting that deployed worker has higher limits?

L

Luka•2/14/24, 4:00 PM

Haha, I did that, Nora. I got same errors. But I am, nevertheless, curious to learn how much different they are

M

Max (@rozenmd)OP•2/14/24, 4:05 PM

the Worker that runs when you run

wrangler dev --remote

wrangler dev --remote

doesn't have the same performance characteristics as a deployed Worker (which runs across Cloudflare's network)

Original message was deleted

L

Luka•2/14/24, 4:17 PM

I think it worked! It took 3 minutes for ~550MB and ~115k rows. Gonna test on local sqlite now

L

Luka•2/14/24, 4:27 PM

It seems to be OK! Thank youuuu so much man. I have no idea how to express how much stress you lifted from me, @Alexandеr . Thank you for it. I think you also deserve updating your

README.md

README.md

that it can potentially handle big DBs.

I will make PR later to Cloudflare docs if it's not there yet amongst other community tools.

And thank you Nora for your willingness and support along the way. It takes a great deal to maintain the package.

S

Sébastien Morel•2/14/24, 5:42 PM

hello here,this change about

.run

.run

I have updated the types and wrangler and I still see that

run

run

returns a D1Response. What I am missing?

S

Santosh•2/14/24, 7:32 PM

When I start wrangler dev, I am not able to resolve a table that I have created by using a local command this way:

 wrangler d1 execute TestDB --local --file=./sql/sequence.sql

 wrangler d1 execute TestDB --local --file=./sql/sequence.sql

.

Uncaught (async) Error: D1_ERROR: SqliteError: no such table:

Uncaught (async) Error: D1_ERROR: SqliteError: no such table:

S

Santosh•2/14/24, 7:33 PM

What could I be doing wrong? Would I have to tell wrangler something abut using my local db ?

S

Santosh•2/14/24, 7:36 PM

I even ran a pragma and I could the fields of the table. When I start wrangler, it even shows my test db:

- D1 Databases:
  - DB: TestDB (c73a0***-

- D1 Databases:
  - DB: TestDB (c73a0***-

S

Santosh•2/14/24, 9:05 PM

const table = new Model(
{
D1Orm: orm,
tableName: "MyTableName",
primaryKeys: "id",
},

S

Santosh•2/14/24, 9:06 PM

Using the orm tool

S

Santosh•2/14/24, 9:09 PM

Is there a concept of preview_database_id that I need to mention inside wrangler? If so, how do I get it?

S

Santosh•2/14/24, 9:15 PM

Same with env.DB.exec("SELECT * FROM MyTableName");

S

Santosh•2/14/24, 9:15 PM

The table is out there

S

Santosh•2/14/24, 9:29 PM

yes I did .

S

Santosh•2/14/24, 9:30 PM

S

Santosh•2/14/24, 9:31 PM

When I run this it works:

wrangler d1 execute TestDB --local --command="SELECT * FROM MyTable"

wrangler d1 execute TestDB --local --command="SELECT * FROM MyTable"

wrangler d1 execute TestDB --local --command="SELECT * FROM MyTable"

wrangler d1 execute TestDB --local --command="SELECT * FROM MyTable"

S

Santosh•2/14/24, 9:31 PM

So it knows the local table exists

SShyim Is the experimental backend enabled by default? :Thinktastic: https://blog.cloud...

M

Max (@rozenmd)OP•2/14/24, 9:49 PM

yes, and will be the only backend eventually

S

Santosh•2/14/24, 10:51 PM

Can someone help me? Not sure how I can do dev work if I cant access local table?

S

Santosh•2/14/24, 10:58 PM

wrangler dev doesnt seem to be able to locate local tables at all. I tried both case (CamelCase, All lowercase) as well for the table names

Ɽ

ⱤØ₥₳₦•2/15/24, 2:02 AM

What will happen in case of pop outage? Like if my D1 is in Singapore and that pop faces down time or scheduled maintenance

ⱤⱤØ₥₳₦What will happen in case of pop outage? Like if my D1 is in Singapore and that p...

C

Chaika•2/15/24, 2:16 AM

D1 runs on Durable Objects. Durable Object Storage is replicated to mutiple nearby datacenters (also why DOs/D1 is only in a few locations). If a specific colo/pop goes down, it should just failover/recreate in another, and it'd have the storage to do so.
Some of that might be slightly different with how D1 stores things, but should be generally the same

G

gratitude3908•2/15/24, 6:14 AM

I'm working on a new startup and have used various cloudflare services for over a decade (huge fan) . I'm looking at database solutions and am considering d1 for a laravel application. My previous application had 12,000 batch requests per second and approximately 500GB of data. We used elastic caching etc. in front. Looking at the pricing I cannot figure out how to think about it. Storage price seems high by itself. Does the group have any suggestions or thoughts?

Ggratitude3908 I'm working on a new startup and have used various cloudflare services for over ...

M

Max (@rozenmd)OP•2/15/24, 6:29 AM

for that kind of load I would use Postgres and Hyperdrive

SShyim Can I see somewhere where my D1 is hosted? I want to debug why the performance i...

M

Max (@rozenmd)OP•2/15/24, 9:26 AM

wrangler d1 info <dbname>

wrangler d1 info <dbname>

wrangler d1 info <dbname>

wrangler d1 info <dbname>

A

amigoz•2/15/24, 9:31 AM

Hi
I am having a specific issue with D1.

I am able to read data from the D1 console/dashboard. But request from worker returns no data. It just says success

{"success":true,"meta":{"served_by":"v3-prod","duration":0.1461,"changes":0,"last_row_id":0,"changed_db":false,"size_after":3919872,"rows_read":10,"rows_written":0}}

Writing data works. But reading does not.

Can anyone help?

A

amigoz•2/15/24, 9:32 AM

Able to read from the console

A

amigoz•2/15/24, 9:44 AM

yes using .run()

A

amigoz•2/15/24, 9:45 AM

Now it works. Thanks a lot.

A

amigoz•2/15/24, 9:46 AM

What about insert? We are still using .run() and it works. Is that also going to change

Aamigoz What about insert? We are still using .run() and it works. Is that also going to...

A

amigoz•2/15/24, 9:49 AM

Got it. Just looked at the doc again. Thanks

B

bun•2/15/24, 11:31 AM

what does d1 database need to be production ready ?

Ɽ

ⱤØ₥₳₦•2/15/24, 11:37 AM

I'm using worker, how do i exactly know the time D1 took to solve the query

I

impairedFelon•2/15/24, 11:37 AM

Check meta that query returns.

I

impairedFelon•2/15/24, 11:37 AM

Object {
  served_by: v3-prod,
  duration: 0.2097,
  changes: 0,
  last_row_id: 0,
  changed_db: false
  ...
}

Object {
  served_by: v3-prod,
  duration: 0.2097,
  changes: 0,
  last_row_id: 0,
  changed_db: false
  ...
}

Object {
  served_by: v3-prod,
  duration: 0.2097,
  changes: 0,
  last_row_id: 0,
  changed_db: false
  ...
}

Object {
  served_by: v3-prod,
  duration: 0.2097,
  changes: 0,
  last_row_id: 0,
  changed_db: false
  ...
}

I

impairedFelon•2/15/24, 11:38 AM

const { results, meta } = await stmt.bind(email).all();

const { results, meta } = await stmt.bind(email).all();

const { results, meta } = await stmt.bind(email).all();

const { results, meta } = await stmt.bind(email).all();

I

impairedFelon•2/15/24, 12:09 PM

After running this.

npx wrangler@latest d1 insights <myDB> --sortBy reads --timePeriod 7d

npx wrangler@latest d1 insights <myDB> --sortBy reads --timePeriod 7d

npx wrangler@latest d1 insights <myDB> --sortBy reads --timePeriod 7d

npx wrangler@latest d1 insights <myDB> --sortBy reads --timePeriod 7d

I find out this.

{
    "query": "SELECT (SELECT count(*) FROM \"table_one\") as \"table_one\",(SELECT count(*) FROM \"table_two\") as \"table_two\",(SELECT count(*) FROM \"table_three\") as \"table_three\",(SELECT count(*) FROM \"table_four\") as \"table_four\",(SELECT count(*) FROM \"table_five\") as \"table_five\",(SELECT count(*) FROM \"table_six\") as \"table_six\",(SELECT count(*) FROM \"table_seven\") as \"table_seven\",(SELECT count(*) FROM \"table_eight\") as \"table_eight\";",
    "avgRowsRead": 396035,
    "totalRowsRead": 792070,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 18.983150000000002,
    "totalDurationMs": 37.966300000000004,
    "numberOfTimesRun": 2
  },

{
    "query": "SELECT (SELECT count(*) FROM \"table_one\") as \"table_one\",(SELECT count(*) FROM \"table_two\") as \"table_two\",(SELECT count(*) FROM \"table_three\") as \"table_three\",(SELECT count(*) FROM \"table_four\") as \"table_four\",(SELECT count(*) FROM \"table_five\") as \"table_five\",(SELECT count(*) FROM \"table_six\") as \"table_six\",(SELECT count(*) FROM \"table_seven\") as \"table_seven\",(SELECT count(*) FROM \"table_eight\") as \"table_eight\";",
    "avgRowsRead": 396035,
    "totalRowsRead": 792070,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 18.983150000000002,
    "totalDurationMs": 37.966300000000004,
    "numberOfTimesRun": 2
  },

{
    "query": "SELECT (SELECT count(*) FROM \"table_one\") as \"table_one\",(SELECT count(*) FROM \"table_two\") as \"table_two\",(SELECT count(*) FROM \"table_three\") as \"table_three\",(SELECT count(*) FROM \"table_four\") as \"table_four\",(SELECT count(*) FROM \"table_five\") as \"table_five\",(SELECT count(*) FROM \"table_six\") as \"table_six\",(SELECT count(*) FROM \"table_seven\") as \"table_seven\",(SELECT count(*) FROM \"table_eight\") as \"table_eight\";",
    "avgRowsRead": 396035,
    "totalRowsRead": 792070,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 18.983150000000002,
    "totalDurationMs": 37.966300000000004,
    "numberOfTimesRun": 2
  },

{
    "query": "SELECT (SELECT count(*) FROM \"table_one\") as \"table_one\",(SELECT count(*) FROM \"table_two\") as \"table_two\",(SELECT count(*) FROM \"table_three\") as \"table_three\",(SELECT count(*) FROM \"table_four\") as \"table_four\",(SELECT count(*) FROM \"table_five\") as \"table_five\",(SELECT count(*) FROM \"table_six\") as \"table_six\",(SELECT count(*) FROM \"table_seven\") as \"table_seven\",(SELECT count(*) FROM \"table_eight\") as \"table_eight\";",
    "avgRowsRead": 396035,
    "totalRowsRead": 792070,
    "avgRowsWritten": 0,
    "totalRowsWritten": 0,
    "avgDurationMs": 18.983150000000002,
    "totalDurationMs": 37.966300000000004,
    "numberOfTimesRun": 2
  },

What is this, i don't have this query in any part of my code.

+1 to deploying it, `wrangler dev`'s Workers !== real Workers

Similar Threads

Similar Threads

Similar Threads