we'll add a note to replace these here in the meanwhile https://developers.cloudflare.com/d1/build-d

we'll add a note to replace these here in the meanwhile https://developers.cloudflare.com/d1/build-databases/import-data/#convert-sqlite-database-files

wesbos•2/28/24, 9:21 PM

it worked! Thank you

MMax (@rozenmd)we'll add a note to replace these here in the meanwhile https://developers.cloud...

Isaac McFadyen•2/28/24, 10:26 PM

The COMMIT part is under https://developers.cloudflare.com/d1/build-databases/import-data/#troubleshooting but yeah, good to move up a section too

matt•2/28/24, 11:27 PM

can you query d1 from outside a cloudflare worker?

matt•2/29/24, 12:05 AM

thanks @geelen for my use case I have a kafka topic that I want to consume from & then write results to d1. I guess my best option is to run a cloudflare worker on a cron trigger every 1 minute to consume, process & write to d1

matt•2/29/24, 12:18 AM

so this is coming off the back of a realtime streams system (bytewax) so I guess I would create a new flow that pulls from the kafka topic and then writes to d1

matt•2/29/24, 12:19 AM

and then deploy it onto my k8s cluster. im building a zapier-like system so d1 would be one of the potential places where data is exported

matt•2/29/24, 12:21 AM

I think I might do that, then it'll also reduce the coupling between the service & the cloudflare environment as it'll just need to know url + api key

matt•2/29/24, 12:22 AM

i think i might avoid cron as there's a chance in future i need to work on making this ultra-low-latency

matt•2/29/24, 12:23 AM

thanks for the advice

HHyperLapse Hi! I want to ask little but important question for me :) According https://deve...

Max (@rozenmd)OP•2/29/24, 12:42 PM

D1 is built to work to the strengths of Cloudflare's network, so we can't offer extremely large databases while also loading them quickly. It's significantly easier for us to offer millions of DBs with a size limit instead.

The limit is currently 2GB and we're working towards making it 10GB - but D1 will require a change in how you think about your project's data if you're looking to build a big platform. Instead of storing all of your data in a single DB, you would store each customer's data in their own DB (and we're building tooling to make it easier to dynamically access multiple DBs in a single Worker without redeploying)

MMax (@rozenmd)D1 is built to work to the strengths of Cloudflare's network, so we can't offer ...

João Castro•2/29/24, 2:50 PM

Are you also building tooling to help with executing migrations on a group of databases?

JJoão Castro Are you also building tooling to help with executing migrations on a group of da...

Max (@rozenmd)OP•2/29/24, 2:50 PM

as a follow-up to dynamic bindings, yes

itsmatteomanf•2/29/24, 3:07 PM

I'm getting "error": "D1_ERROR: D1 is overloaded. Requests queued for too long.""error": "D1_ERROR: D1 is overloaded. Requests queued for too long." on delete queries, some do succeed, some don't but I technically don't know unless I check row counts...

itsmatteomanf•2/29/24, 3:08 PM

It's a lot of rows (100s of thousands), but with the only filtering clause on a column with an index...

Iitsmatteomanf I'm getting `"error": "D1_ERROR: D1 is overloaded. Requests queued for too long....

Max (@rozenmd)OP•2/29/24, 3:34 PM

whats the db id again?

MMax (@rozenmd)whats the db id again?

itsmatteomanf•2/29/24, 3:35 PM

546787dd-35d5-4822-9b50-36fa0a4310c2

546787dd-35d5-4822-9b50-36fa0a4310c2

Iitsmatteomanf `546787dd-35d5-4822-9b50-36fa0a4310c2`

Max (@rozenmd)OP•2/29/24, 3:40 PM

were you doing a delete on top of the regular task that happens every hour?

MMax (@rozenmd)were you doing a delete on top of the regular task that happens every hour?

itsmatteomanf•2/29/24, 3:41 PM

Yes, I was... might have been just after it started, to be fair, I didn't check the time.

Max (@rozenmd)OP•2/29/24, 3:41 PM

looks like thats what tipped it into overloading

itsmatteomanf•2/29/24, 3:41 PM

I see...

MMax (@rozenmd)looks like thats what tipped it into overloading

itsmatteomanf•2/29/24, 3:51 PM

The weird thing is that the error was returned, but the delete succeded.

KJolibois•2/29/24, 4:14 PM

I'm working on d1 to build something out with a friend, I added him to my account (e.g kj@emailprovider.com) and gave him (e.g kfriend@emailprovider.net) workers permission for all domains. At first he could not see the databases when logged in, whenever he clicked on d1 it kept sending him to the overview page. After I sent him a link to the page I was using (eg https://dash.cloudflare.com/{D1-ID-HERE}/workers/d1) he was able to see the database, but the account name on the top left changed from his email to mine. We were able to collaborate just fine. There was probably a step we missed where he needed to change over from his default account to mine ,but I thought I should ping just in case it was unexpected behavior. Thanks for all the great work on d1. I'm loving it so far .

Iitsmatteomanf The weird thing is that the error was returned, but the delete succeded.

Tony Leung•2/29/24, 4:26 PM

but not all of it right? I guess that you are batching a lot of DELETE commands, not delete by ID. According to the document, I think cf implements batch using a loop, trying to find a gap between other queries to run commands. If the batch does not success after a period of time, it will flag the batch as overloaded and returns the failed command index (as shown on the batch array). Since queries are run sequentially and commit on each success one then that's why your batch failed but some of them are executed.

Tony Leung•2/29/24, 4:27 PM

As a mysql user, on some large delete execution plan, i tend to select all the record ID which I want to delete first then delete by ID later

TTony Leung but not all of it right? I guess that you are batching a lot of DELETE commands,...

itsmatteomanf•2/29/24, 4:28 PM

So, afaik, batches are transactions behind the scenes, so either they all run or none do.
In any case, here, I was doing a DELETE FROM table WHERE datetime < datetime('yyyy-mm-dd')DELETE FROM table WHERE datetime < datetime('yyyy-mm-dd'), single query.

Iitsmatteomanf So, afaik, batches are transactions behind the scenes, so either they all run or...

Tony Leung•2/29/24, 4:29 PM

do you index the datetimedatetime row?

TTony Leung do you index the `datetime` row?

itsmatteomanf•2/29/24, 4:29 PM

Yes, of course

Iitsmatteomanf Yes, of course 🙂

Tony Leung•2/29/24, 4:30 PM

oof so your table must have a million rows XD

TTony Leung oof so your table must have a million rows XD

itsmatteomanf•2/29/24, 4:30 PM

The table was actually 10M, before the deletes, which I did in 14d groups, to avoid too much overloading, but wasn't enough sometimes apaprently :lul:

TTony Leung oof so your table must have a million rows XD

Tony Leung•2/29/24, 4:32 PM

it reminds me of one time I tried to execute 10 or 20 query like that on a mysql table which has 300m rows. The production was down shortly after, the db was not responding . I was using aurora serverless v1 so there is no way to reboot the DB. It hangs for 8 or 9 hrs and costs me about 200 - 300$

Iitsmatteomanf The table was actually 10M, before the deletes, which I did in 14d groups, to av...

Tony Leung•2/29/24, 4:33 PM

oh...

Tony Leung•2/29/24, 4:33 PM

I don't think a cluster of oracle DB could handle that...

TTony Leung oh...

itsmatteomanf•2/29/24, 4:36 PM

To be fair, I was told to go ham on it and it does sustain 1000 write queries per minute, along with 100 read queries per minute. Optimized queries, most are single or a few rows each, but still.

itsmatteomanf•2/29/24, 4:37 PM

No sharding, multiple Workers at the same time, it's pretty good for what it is.

Tony Leung•2/29/24, 4:37 PM

I think you should select the ID first, then mark the record as "is_deleted" then actual delete them on low-traffic hour

Iitsmatteomanf To be fair, I was told to go ham on it and it does sustain 1000 write queries pe...

Tony Leung•2/29/24, 4:37 PM

that's an impressive number with a price next to 0$

TTony Leung that's an impressive number with a price next to 0$

itsmatteomanf•2/29/24, 4:38 PM

Definitely!

TTony Leung I think you should select the ID first, then mark the record as "is_deleted" the...

itsmatteomanf•2/29/24, 4:38 PM

It's mostly manual deletes now, I should just look at the time and do it not at the start of the task that spikes up in query volume :lul:

Tony Leung•2/29/24, 4:38 PM

well at least cf will protect me from ddos attack which would cost me 104k of hard earned money

Iitsmatteomanf It's mostly manual deletes now, I should just look at the time and do it not at ...

Tony Leung•2/29/24, 4:41 PM

I usually schedule deletion on 1 to 3am

Original message was deleted

Tony Leung•2/29/24, 4:57 PM

my customer's timezone, since my app are only used in some countries with adjacent timezones so at that time most of them are sleeping

Tony Leung•2/29/24, 4:58 PM

(or at least that's what it shown on the traffic chart XD)

we'll add a note to replace these here in the meanwhile https://developers.cloudflare.com/d1/build-d

Similar Threads