Cloudflare Developers•2y ago

How to predefine location for D1

ⱤⱤØ₥₳₦How to predefine location for D1

Matt Silverlock•2/14/24, 2:48 PM

https://developers.cloudflare.com/d1/configuration/data-location/

Cloudflare Docs

Data location · Cloudflare D1 docs

Learn how the location of data stored in D1 is determined, including where the leader is placed and how you optimize that location based on your …

Ian•2/14/24, 2:54 PM

will dynamic binding be part of GA release by April ?

MMax (@rozenmd).sqlite export is being specced out but is unlikely to arrive within the month, ...

Luka•2/14/24, 2:58 PM

Unfortunately I keep hitting the limits either memory/max execution time or sub requests. I opened ticket at cloudflare if they can lift the limits but pretty much it always errors out with no information. I really feel hopeless on that one other than manually transferring the rows.

If you would feel kind enough to look through and share any ideas on the issue, I'd really appreciate that!
https://github.com/nora-soderlund/cloudflare-d1-backups/issues/9

GitHub

Errors: Http 500 - CF 1101 // Http 503 - CF 1105 // Http 504 · Issu...

Hi again. I'm back at it once again trying to export a 450 MB-large database. My own worker uses Hono and I would constantly see the Inspector command exceeded memory limits errors. I've de...

MMatt Silverlock https://developers.cloudflare.com/d1/configuration/data-location/

Ɽ

ⱤØ₥₳₦OP•2/14/24, 3:13 PM

Dayum, now I'm legally blind

. Thanks for the resource

Original message was deleted

Max (@rozenmd)•2/14/24, 3:22 PM

+1 to deploying it,

wrangler dev

wrangler dev

's Workers !== real Workers

Original message was deleted

Luka•2/14/24, 3:57 PM

Yes

Original message was deleted

Luka•2/14/24, 3:58 PM

That's great. But there's this one table that has around 90k rows and I suspect it being a few hundred MBs

MMax (@rozenmd)+1 to deploying it, `wrangler dev`'s Workers !== real Workers

Luka•2/14/24, 3:59 PM

to remove any ambiguity, are you talking about worker dev --remoteworker dev --remote?

LLuka to remove any ambiguity, are you talking about `worker dev --remote`? 🙂

Max (@rozenmd)•2/14/24, 3:59 PM

yes

MMax (@rozenmd)yes

Luka•2/14/24, 3:59 PM

so you're suggesting that deployed worker has higher limits?

Luka•2/14/24, 4:00 PM

Haha, I did that, Nora. I got same errors. But I am, nevertheless, curious to learn how much different they are

Max (@rozenmd)•2/14/24, 4:05 PM

the Worker that runs when you run

wrangler dev --remote

wrangler dev --remote

doesn't have the same performance characteristics as a deployed Worker (which runs across Cloudflare's network)

Original message was deleted

Luka•2/14/24, 4:17 PM

I think it worked! It took 3 minutes for ~550MB and ~115k rows. Gonna test on local sqlite now

Luka•2/14/24, 4:27 PM

It seems to be OK! Thank youuuu so much man. I have no idea how to express how much stress you lifted from me, @Alexandеr . Thank you for it. I think you also deserve updating your README.mdREADME.md that it can potentially handle big DBs.

I will make PR later to Cloudflare docs if it's not there yet amongst other community tools.

And thank you Nora for your willingness and support along the way. It takes a great deal to maintain the package.

Sébastien Morel•2/14/24, 5:42 PM

hello here,this change about

.run

.run

I have updated the types and wrangler and I still see that

run

run

returns a D1Response. What I am missing?

Santosh•2/14/24, 7:32 PM

When I start wrangler dev, I am not able to resolve a table that I have created by using a local command this way:

 wrangler d1 execute TestDB --local --file=./sql/sequence.sql

 wrangler d1 execute TestDB --local --file=./sql/sequence.sql

Uncaught (async) Error: D1_ERROR: SqliteError: no such table:

Uncaught (async) Error: D1_ERROR: SqliteError: no such table:

Santosh•2/14/24, 7:33 PM

What could I be doing wrong? Would I have to tell wrangler something abut using my local db ?

Santosh•2/14/24, 7:36 PM

I even ran a pragma and I could the fields of the table. When I start wrangler, it even shows my test db:

- D1 Databases:
  - DB: TestDB (c73a0***-

- D1 Databases:
  - DB: TestDB (c73a0***-

Santosh•2/14/24, 9:05 PM

const table = new Model(
{
D1Orm: orm,
tableName: "MyTableName",
primaryKeys: "id",
},

Santosh•2/14/24, 9:06 PM

Using the orm tool

Santosh•2/14/24, 9:09 PM

Is there a concept of preview_database_id that I need to mention inside wrangler? If so, how do I get it?

Santosh•2/14/24, 9:15 PM

Same with env.DB.exec("SELECT * FROM MyTableName");

Santosh•2/14/24, 9:15 PM

The table is out there

Santosh•2/14/24, 9:29 PM

yes I did .

Santosh•2/14/24, 9:30 PM

Santosh•2/14/24, 9:31 PM

When I run this it works:

wrangler d1 execute TestDB --local --command="SELECT * FROM MyTable"

wrangler d1 execute TestDB --local --command="SELECT * FROM MyTable"

Santosh•2/14/24, 9:31 PM

So it knows the local table exists

SShyim Is the experimental backend enabled by default? :Thinktastic: https://blog.cloud...

Max (@rozenmd)•2/14/24, 9:49 PM

yes, and will be the only backend eventually

Santosh•2/14/24, 10:51 PM

Can someone help me? Not sure how I can do dev work if I cant access local table?

Santosh•2/14/24, 10:58 PM

wrangler dev doesnt seem to be able to locate local tables at all. I tried both case (CamelCase, All lowercase) as well for the table names

Ɽ

ⱤØ₥₳₦OP•2/15/24, 2:02 AM

What will happen in case of pop outage? Like if my D1 is in Singapore and that pop faces down time or scheduled maintenance

ⱤⱤØ₥₳₦What will happen in case of pop outage? Like if my D1 is in Singapore and that p...

Chaika•2/15/24, 2:16 AM

D1 runs on Durable Objects. Durable Object Storage is replicated to mutiple nearby datacenters (also why DOs/D1 is only in a few locations). If a specific colo/pop goes down, it should just failover/recreate in another, and it'd have the storage to do so.
Some of that might be slightly different with how D1 stores things, but should be generally the same

gratitude3908•2/15/24, 6:14 AM

I'm working on a new startup and have used various cloudflare services for over a decade (huge fan) . I'm looking at database solutions and am considering d1 for a laravel application. My previous application had 12,000 batch requests per second and approximately 500GB of data. We used elastic caching etc. in front. Looking at the pricing I cannot figure out how to think about it. Storage price seems high by itself. Does the group have any suggestions or thoughts?

Ggratitude3908 I'm working on a new startup and have used various cloudflare services for over ...

Max (@rozenmd)•2/15/24, 6:29 AM

for that kind of load I would use Postgres and Hyperdrive

SShyim Can I see somewhere where my D1 is hosted? I want to debug why the performance i...

Max (@rozenmd)•2/15/24, 9:26 AM

wrangler d1 info <dbname>wrangler d1 info <dbname>

amigoz•2/15/24, 9:31 AM

Hi
I am having a specific issue with D1.

I am able to read data from the D1 console/dashboard. But request from worker returns no data. It just says success

{"success":true,"meta":{"served_by":"v3-prod","duration":0.1461,"changes":0,"last_row_id":0,"changed_db":false,"size_after":3919872,"rows_read":10,"rows_written":0}}

Writing data works. But reading does not.

Can anyone help?

amigoz•2/15/24, 9:32 AM

Able to read from the console