So, afaik, batches are transactions behind the scenes, so either they all run or none do. In any cas

So, afaik, batches are transactions behind the scenes, so either they all run or none do.
In any case, here, I was doing a

DELETE FROM table WHERE datetime < datetime('yyyy-mm-dd')

DELETE FROM table WHERE datetime < datetime('yyyy-mm-dd')

DELETE FROM table WHERE datetime < datetime('yyyy-mm-dd')

DELETE FROM table WHERE datetime < datetime('yyyy-mm-dd'), single query.

Iitsmatteomanf So, afaik, batches are transactions behind the scenes, so either they all run or...

Tony Leung•2/29/24, 4:29 PM

do you index the datetimedatetime row?

TTony Leung do you index the `datetime` row?

itsmatteomanfOP•2/29/24, 4:29 PM

Yes, of course

Iitsmatteomanf Yes, of course 🙂

Tony Leung•2/29/24, 4:30 PM

oof so your table must have a million rows XD

TTony Leung oof so your table must have a million rows XD

itsmatteomanfOP•2/29/24, 4:30 PM

The table was actually 10M, before the deletes, which I did in 14d groups, to avoid too much overloading, but wasn't enough sometimes apaprently :lul:

TTony Leung oof so your table must have a million rows XD

Tony Leung•2/29/24, 4:32 PM

it reminds me of one time I tried to execute 10 or 20 query like that on a mysql table which has 300m rows. The production was down shortly after, the db was not responding . I was using aurora serverless v1 so there is no way to reboot the DB. It hangs for 8 or 9 hrs and costs me about 200 - 300$

Iitsmatteomanf The table was actually 10M, before the deletes, which I did in 14d groups, to av...

Tony Leung•2/29/24, 4:33 PM

oh...

Tony Leung•2/29/24, 4:33 PM

I don't think a cluster of oracle DB could handle that...

TTony Leung oh...

itsmatteomanfOP•2/29/24, 4:36 PM

To be fair, I was told to go ham on it and it does sustain 1000 write queries per minute, along with 100 read queries per minute. Optimized queries, most are single or a few rows each, but still.

itsmatteomanfOP•2/29/24, 4:37 PM

No sharding, multiple Workers at the same time, it's pretty good for what it is.

Tony Leung•2/29/24, 4:37 PM

I think you should select the ID first, then mark the record as "is_deleted" then actual delete them on low-traffic hour

Iitsmatteomanf To be fair, I was told to go ham on it and it does sustain 1000 write queries pe...

Tony Leung•2/29/24, 4:37 PM

that's an impressive number with a price next to 0$

TTony Leung that's an impressive number with a price next to 0$

itsmatteomanfOP•2/29/24, 4:38 PM

Definitely!

TTony Leung I think you should select the ID first, then mark the record as "is_deleted" the...

itsmatteomanfOP•2/29/24, 4:38 PM

It's mostly manual deletes now, I should just look at the time and do it not at the start of the task that spikes up in query volume :lul:

Tony Leung•2/29/24, 4:38 PM

well at least cf will protect me from ddos attack which would cost me 104k of hard earned money

Iitsmatteomanf It's mostly manual deletes now, I should just look at the time and do it not at ...

Tony Leung•2/29/24, 4:41 PM

I usually schedule deletion on 1 to 3am

Original message was deleted

Tony Leung•2/29/24, 4:57 PM

my customer's timezone, since my app are only used in some countries with adjacent timezones so at that time most of them are sleeping

Tony Leung•2/29/24, 4:58 PM

(or at least that's what it shown on the traffic chart XD)

TTony Leung well at least cf will protect me from ddos attack which would cost me 104k of ha...

Tony Leung•2/29/24, 5:20 PM

btw it actually happened with a netlify user https://redd.it/1b14bty

From the webdev community on Reddit

Explore this post and more from the webdev community

Unsmart•2/29/24, 7:00 PM

afaik its just standard sqlite, but you should be able to create foreign keys if thats what you mean from that

SLW•2/29/24, 7:07 PM

hey guys, How do I integrate prisma with D1?:NotLikeThis:

kian•2/29/24, 7:10 PM

that's basically the only thing taken from Turso

kian•2/29/24, 7:10 PM

kian•2/29/24, 7:10 PM

just the rows read/written stuff

kian•2/29/24, 7:11 PM

it's built from sqlite source still

Bbun ok but why does this not work ``` await this.db.batch([ this.db.insert(bl...

KJolibois•2/29/24, 8:54 PM

where is .insert from? I don't see it in the docs. what is blobs? is there a way to map classes to tables without using prepare and run or are you using an orm on top of d1?

KKJolibois where is .insert from? I don't see it in the docs. what is blobs? is there a wa...

bun•2/29/24, 9:00 PM

It’s fixed. Was my issue

SSLW hey guys, How do I integrate prisma with D1?:NotLikeThis:

Max (@rozenmd)•2/29/24, 9:26 PM

you can't use prisma with d1 yet

Hhannes So my understanding from some workerd commits is that D1 is built on top of http...

Max (@rozenmd)•2/29/24, 9:28 PM

rows_read/rows_written was the one feature we took from libsql, the rest is security-mode sqlite

Bbun It’s fixed. Was my issue

KJolibois•2/29/24, 9:30 PM

I'm still curious as to how you called .insert, can you link me to where they go over that? I don't see that method here https://developers.cloudflare.com/d1/build-databases/query-databases/

Cloudflare Docs

Query databases · Cloudflare D1 docs

This guide documents D1’s client API and how to query D1 from Cloudflare Workers, how D1 maps types from JavaScript/TypeScript and SQL, and common …

KKJolibois I'm still curious as to how you called .insert, can you link me to where they go...

bun•2/29/24, 9:32 PM

Its drizzle orm

KJolibois•2/29/24, 9:33 PM

thank you @nora and @bun

Ggeelen sadly no. Out of interest, what's your use-case?

Tony Leung•3/1/24, 4:37 AM

i guess that he wants to terminate the execution if it does not finish online. In mysql they have query hint like:

SELECT /*+ MAX_EXECUTION_TIME(1000) */ --in milliseconds * FROM table

SELECT /*+ MAX_EXECUTION_TIME(1000) */ --in milliseconds * FROM table

Tony Leung•3/1/24, 4:47 AM

since workers only lives for 30 seconds per fetch invocation, i guess he want it failed rather than worker eviction to raise a reasonable error to the user, or to do something else

TTony Leung since workers only lives for 30 seconds per fetch invocation, i guess he want it...

Hello, I’m Allie!•3/1/24, 5:52 AM

That’s 30 seconds of CPU time, not wall time

Hello, I’m Allie!•3/1/24, 5:52 AM

So even if the response takes longer than 30 seconds, it should still complete

HHello, I’m Allie!That’s 30 seconds of CPU time, not wall time

Tony Leung•3/1/24, 5:53 AM

Oh that's new, i thought bundle workers only have 30s off wall time

TTony Leung Oh that's new, i thought bundle workers only have 30s off wall time

Hello, I’m Allie!•3/1/24, 5:54 AM

Nope:

HHello, I’m Allie!Nope:

Tony Leung•3/1/24, 5:54 AM

Oh nice!

Tony Leung•3/1/24, 5:54 AM

Thanks

Tony Leung•3/1/24, 5:55 AM

Didn't know that they dont limit wall time. I remembered that in a doc workers will have a high chance of eviction if they run pass 30s

TTony Leung Didn't know that they dont limit wall time. I remembered that in a doc workers w...

Hello, I’m Allie!•3/1/24, 6:09 AM

That one is sort of true. As you pass the 30 second mark, your chance of eviction goes up, but I wouldn’t say it is high…

Aalexblokh hey, D1 ninjas! is there a way to use https://developers.cloudflare.com/api/ope...

Max (@rozenmd)•3/1/24, 11:02 AM

not via that API, no, but you could expose a Worker as your own custom HTTP API? https://github.com/elithrar/http-api-d1-example

GitHub

GitHub - elithrar/http-api-d1-example: An example HTTP API for Clou...

An example HTTP API for Cloudflare's D1 database: https://developers.cloudflare.com/d1/

elithrar/http-api-d1-example

So, afaik, batches are transactions behind the scenes, so either they all run or none do. In any cas

Similar Threads

Similar Threads

Similar Threads