Hello, I tried to migrate a large amount of data from postgres into d1 and I got `please wait and co

Hello, I tried to migrate a large amount of data from postgres into d1 and I got

please wait and consider throttling your request speed [code: 971]

please wait and consider throttling your request speed [code: 971]

please wait and consider throttling your request speed [code: 971]

please wait and consider throttling your request speed [code: 971]. What would be the best way to create a large amount (100k rows) in d1?

WWcaleNieWolny Hello, I tried to migrate a large amount of data from postgres into d1 and I got...

Matt Silverlock•1/3/24, 6:38 AM

How are you attempting to migrate? You should batch your inserts - there’s no reason you can’t insert 10k+ rows per statement, or more, depending on overall width.

WcaleNieWolnyOP•1/3/24, 7:46 AM

What do you mean by batch my statements?

WcaleNieWolnyOP•1/3/24, 7:47 AM

I use the http api and I do 1 call per row

WcaleNieWolnyOP•1/3/24, 7:50 AM

How should I batch this?

WWcaleNieWolny How should I batch this?

Matt Silverlock•1/3/24, 8:35 AM

Pass multiple statements in one API call, separated by a semicolon. You’re being rate limited by the HTTP API, not D1 itself.

MMatt Silverlock Pass multiple statements in one API call, separated by a semicolon. You’re being...

WcaleNieWolnyOP•1/3/24, 8:45 AM

Oh, I did not know that. Thanks ^^

Original message was deleted

Max (@rozenmd)•1/3/24, 12:15 PM

it's a bug, if you want data, use .all()

MMatt Silverlock Pass multiple statements in one API call, separated by a semicolon. You’re being...

WcaleNieWolnyOP•1/3/24, 1:57 PM

Error: A prepared SQL statement must contain only one statement.Error: A prepared SQL statement must contain only one statement.

WcaleNieWolnyOP•1/3/24, 2:14 PM

https://gist.github.com/WcaleNieWolny/3510515be7145411d3b3cc93ccd0c433https://gist.github.com/WcaleNieWolny/3510515be7145411d3b3cc93ccd0c433

WcaleNieWolnyOP•1/3/24, 2:14 PM

Essentially I read data from postgrs -> convert it into an insert statement and then I try to execute this in d1

WcaleNieWolnyOP•1/3/24, 2:15 PM

the problem is that I get rate limited - it was suggested I "Pass multiple statements in one API call, separated by a semicolon" - that did not work as I use a prepared statement

WWcaleNieWolny https://gist.github.com/WcaleNieWolny/3510515be7145411d3b3cc93ccd0c433https://gi...

Isaac McFadyen•1/3/24, 2:15 PM

We're not going to be able to debug with that unfortunately, that won't show us the actual SQL being executed.

WcaleNieWolnyOP•1/3/24, 2:16 PM

WcaleNieWolnyOP•1/3/24, 2:16 PM

here is the sql being executed

Isaac McFadyen•1/3/24, 2:16 PM

So the first thing is that D1 doesn't support BEGIN TRANSACTION

Isaac McFadyen•1/3/24, 2:16 PM

Any statements sent as part of the batch API are automatically executed in a transaction, but otherwise transactions are handled by D1 internally.

WcaleNieWolnyOP•1/3/24, 2:17 PM

right - I only have https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database available - will this work as batch there?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

WcaleNieWolnyOP•1/3/24, 2:18 PM

how would you like me to send a batch statement using this http api?

WcaleNieWolnyOP•1/3/24, 2:22 PM

right - so it also does not support the batch api

WWcaleNieWolny right - so it also does not support the batch api

Matt Silverlock•1/3/24, 2:54 PM

Sorry, I interpreted that you had a Worker, or were using wrangler to execute a file export. Row by row in the HTTP API has this limitation (for now)

twilson90•1/3/24, 3:06 PM

Trying to setup a d1 database on the remote worker with the following command:

npx wrangler d1 create testnpx wrangler d1 create test

results in

[ERROR] Cannot read properties of undefined (reading 'map')[ERROR] Cannot read properties of undefined (reading 'map')

Using the --local flag works however.

In my wrangler.tml I have:

[[d1_databases]]
binding = "DB"
database_name = "name-of-db"
database_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

[[d1_databases]]
binding = "DB"
database_name = "name-of-db"
database_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

What am I doing wrong?

EDIT:
Oops. Turned out I'd been signed out of cloudflare and I needed to run npx wrangler loginnpx wrangler login again. Odd because it was still letting me deploy. A better error message would have been helpful.

Ttwilson90 Trying to setup a d1 database on the remote worker with the following command: ...

Freeze•1/3/24, 8:19 PM

OMG I have spend 3 hours on this

Rabid•1/3/24, 9:02 PM

It would be a big QOL change to handle pagination of D1 databases in the UI better. So when you click on a db, and go back, you don’t have to navigate to the page you were on again. Right now the URL is unchanged when paging through DBs or DB rows.

ALECKS•1/4/24, 1:25 AM

Is there any guidance on how to run migrations against a local, development, D1 instance?

AALECKS Is there any guidance on how to run migrations against a local, development, D1 ...

Cyb3r-Jak3•1/4/24, 1:30 AM

You just run the migration command with the --local--local flag.

CCyb3r-Jak3 You just run the migration command with the `--local` flag.

ALECKS•1/4/24, 1:31 AM

thank you!

ALECKS•1/4/24, 1:32 AM

What tools are people using to interact with the loca dev db? just the cli wrangler d1 execute [queries]wrangler d1 execute [queries] or is there some other goto, maybe gui, client that is more friendly?

Rubi•1/4/24, 2:43 AM

will I be allowed to access D1 from outside of cloudflare?

Rubi•1/4/24, 2:43 AM

on GA

Unsmart•1/4/24, 2:45 AM

I'd assume it will stay as it is now. If you want to do that you can either use the API, or better: create a worker proxy.

dragon•1/4/24, 4:26 AM

just me once again saying im super impressed with d1 thus far.
12mb for 15K rows....
This is the db behind https://dstif.io, which runs completely on cf workers, pages, and d1.

Hope this year brings some ux improvements - like search and filter and improved pagination

Ddragon just me once again saying im super impressed with d1 thus far. 12mb for 15K rows...

Rubi•1/4/24, 4:54 AM

does the perf & latency good?

RRubi does the perf & latency good?

dragon•1/4/24, 4:56 AM

I havent had any issues with either perf or latency

TigersWay•1/4/24, 4:57 AM

I do regularly (little bit less than once a day) get these errors: D1_ERROR: Network connection lost.D1_ERROR: Network connection lost. or D1_ERROR: D1 storage operation exceeded timeout which caused object to be reset.D1_ERROR: D1 storage operation exceeded timeout which caused object to be reset. with 1k to 2.5k write request per day.
What would be the best way to deal with these? Should I simply try with the obvious "retry on error" logic, or am I not seeing something else?

dragon•1/4/24, 5:03 AM

have you considered using batch statements (aka sql transactions) ?

dragon•1/4/24, 5:05 AM

i know the error is a connection error - but the transaction would avoid partial queries from being completed

Rubi•1/4/24, 5:07 AM

uh, sqlite haven't support geointersect, I always willing to rewrite my Mongo app to use D1

AAaronO Depending on your use-case, you could consider using something like geohashes (h...

Rubi•1/4/24, 5:26 AM

I'm good, gotta stay with MongoDB, all-in-one solution, until cloudflare support the same functionality, then I will definitely move to cloudflare solution

Rubi•1/4/24, 5:34 AM

I think it will be burden to the memory

AAaronO You mean storage overhead ? Per row ?

Rubi•1/4/24, 5:40 AM

explicitly computing the geohash isn't this mean: loading the geojson to the memory then doing a geospatial processing on it

Rubi•1/4/24, 5:51 AM

basically what I'm doing is https://www.mongodb.com/docs/manual/reference/operator/query/geoIntersects/ and https://www.mongodb.com/docs/manual/reference/operator/query/near/ on >25GB Geojson dataset

Rubi•1/4/24, 5:52 AM

I never specifically learn about geohash, but sure I will take a look

Hello, I tried to migrate a large amount of data from postgres into d1 and I got `please wait and co

Similar Threads

Similar Threads

Similar Threads