i'm executing ~300 requests in parallel to simply run this (300 requests to the worker, each worker

i'm executing ~300 requests in parallel to simply run this (300 requests to the worker, each worker runs the query only once)

const stmt = env.DB.prepare(
    "INSERT OR IGNORE INTO topic_queue (topic_id) VALUES (?)"
).bind(parentTopicId);
await stmt.run();

const stmt = env.DB.prepare(
    "INSERT OR IGNORE INTO topic_queue (topic_id) VALUES (?)"
).bind(parentTopicId);
await stmt.run();

const stmt = env.DB.prepare(
    "INSERT OR IGNORE INTO topic_queue (topic_id) VALUES (?)"
).bind(parentTopicId);
await stmt.run();

const stmt = env.DB.prepare(
    "INSERT OR IGNORE INTO topic_queue (topic_id) VALUES (?)"
).bind(parentTopicId);
await stmt.run();

Ooof2win2 i'm executing ~300 requests in parallel to simply run this (300 requests to the ...

Isaac McFadyen•9/12/24, 3:38 PM

You should probably run in fewer statements, each inserting more than one record. A few requests inserting more than one item will be more efficient than many requests all inserting one item.

oof2win2OP•9/12/24, 3:44 PM

yeah but i run one insert per http request

oof2win2OP•9/12/24, 3:44 PM

cant really optimize that better

Ooof2win2 yeah but i run one insert per http request

Isaac McFadyen•9/12/24, 3:47 PM

Oh I see, I thought you were talking about bulk inserting data but you mean that each request actually only generates 1 row to insert...

bovium•9/12/24, 5:03 PM

Hi I would like to know if there is a way to remove data back in time in the D1 time travel backup databases.

In these days of GDPR it is a very relevant feature that you are able to remove data from your backups.

IIsaac McFadyen Oh I see, I thought you were talking about bulk inserting data but you mean that...

oof2win2OP•9/12/24, 5:09 PM

yeah my bad thats what i mean

ysm•9/13/24, 3:40 AM

How to get metameta object (duration, row_read, row_written) in D1 + Prisma query?

1984 Ford Laser•9/13/24, 7:54 AM

I have a bunch of data in a mongo install that I want to move to D1. For ease of development and saving my sanity, is there any downside to storing all data in D1 as JSON? and just querying the json as necessary? Would have index columns as independent columnds outside of the JSON column. Mainly wondering about performance issues that may arise

11984 Ford Laser I have a bunch of data in a mongo install that I want to move to D1. For ease of...

oof2win2OP•9/13/24, 9:36 AM

use jsonb. you can create indexes on expressions so you can essentially index say a userid field on a json (or something deeply nested as well)

oof2win2OP•9/13/24, 9:36 AM

jsonb because it is stored better and performs better, but even normal json has the ability to be indexed

Original message was deleted

�

𝓈𝒽𝓎•9/13/24, 11:34 AM

does this also work with the 10GB limit per Database?

MMax (@rozenmd)If you're set on storing that data in a single database, we'd recommend using Hy...

�

𝓈𝒽𝓎•9/13/24, 11:35 AM

I thought due to this message it's a hard limit

�

𝓈𝒽𝓎•9/13/24, 11:35 AM

what happens if you reach 10GB?

SS0ns0fAnarchy After days of research in forums, Cloudflare docs etc. I'm still left with sever...

1984 Ford Laser•9/13/24, 12:40 PM

You say 5-6GB per user, what is the data? Sounds like a good candidate for R2 with using D1 to store references to the files on R2

SS0ns0fAnarchy After days of research in forums, Cloudflare docs etc. I'm still left with sever...

Hard@Work•9/13/24, 1:08 PM

IIRC it is a raisable limit

acan•9/13/24, 1:13 PM

To be honest, I am looking forward to the support of ON UPDATE CURRENT_TIMESTAMPON UPDATE CURRENT_TIMESTAMP

�𝓈𝒽𝓎 what happens if you reach 10GB?

Isaac McFadyen•9/13/24, 1:56 PM

Your DB starts returning an error on inserts I believe.

Isaac McFadyen•9/13/24, 1:57 PM

The 10GB is not raisable. The question above was for the account-wide limit across multiple DBs.

DDean accessing D1 via the CF API is slow because it has to route to a core location. ...

nulled•9/13/24, 3:00 PM

Thanks! I was hoping to avoid exactly that, but a confirmation that that is the best way forward and I wasn't missing something obvious helps!

Aacan To be honest, I am looking forward to the support of `ON UPDATE CURRENT_TIMESTAM...

nulled•9/13/24, 3:03 PM

Do you mean an UPDATE trigger that updates a timestamp on a table each time you sent an UPDATE to some table? If so I've been using exactly that for as long as I've been using D1

CREATE TRIGGER trigger_name AFTER UPDATE ON table BEGIN UPDATE another_table SET timestamp=UNIXEPOCH('NOW') WHERE column=some_val; END

CREATE TRIGGER trigger_name AFTER UPDATE ON table BEGIN UPDATE another_table SET timestamp=UNIXEPOCH('NOW') WHERE column=some_val; END

CarmeloCampos•9/13/24, 7:32 PM

if I have two different projects, and I need to use the same database (D1) how can I share the schemas?

conor•9/13/24, 11:12 PM

Similar to the above, if I'm running to workers locally, how can I get them to share the same database?

shixzie•9/14/24, 12:15 AM

are there any d1-related anouncements planned for birthday week?

Ooof2win2 i'm executing ~300 requests in parallel to simply run this (300 requests to the ...

akashra•9/14/24, 4:20 AM

Is there some particular reason you're not using a single prepared statement with multiple param sets called as a single query?

Aakashra Is there some particular reason you're not using a single prepared statement wit...

oof2win2OP•9/14/24, 6:19 AM

i am executing the queries in a new fetch request every time. i literally can’t have multiple params as the requests are independent of each other

Hello, I’m Allie!•9/14/24, 8:58 AM

Point them to the same directory with —persist-to—persist-to

Nnulled Do you mean an UPDATE trigger that updates a timestamp on a table each time you ...

acan•9/14/24, 1:38 PM

I am surprised that D1 supports triggers. It's great.

Aacan I am surprised that D1 supports triggers. It's great.

Puliczek•9/14/24, 5:16 PM

I wish there could be cloudflare worker trigger on D1 table new row.

Puliczek•9/14/24, 5:17 PM

I would like to create new payment/new user -> send notification to telegram/discord.

HHello, I’m Allie!Point them to the same directory with `—persist-to`

conor•9/14/24, 6:09 PM

I tried that, state persists but it doesn't have my migrations applied and doesn't seem to want to apply them (no migrations to apply!). ~~Do I have to use the same flag for the migrations command I wonder?~~

Same flag not applicable to the migrations command and I'm not sure there's an equivalent.

I get:

✘ [ERROR] Implications failed:

   persist-to -> local

✘ [ERROR] Implications failed:

   persist-to -> local

But when I use

--local

--local

instead of persist-topersist-to for migrations I get :white_check_mark: No migrations to apply!:white_check_mark: No migrations to apply!

conor•9/14/24, 6:12 PM

Running wrangler 3.78.0wrangler 3.78.0

conor•9/14/24, 6:21 PM

Ah ok! You need to supply both the

--persist-to

--persist-to

flag and the

--local

--local

flag. I imagined

--persist-to

--persist-to

would imply locallocal because I didn't think it was possible to direct persistence to a path remotely?

Original message was deleted

Puliczek•9/14/24, 6:33 PM

I have 10+ db and apps, I prefer to have one trigger cloudflare worker to do that, without CRON. Or maybe CRON 1hr should be ok, maybe thats the way.

PPuliczek I have 10+ db and apps, I prefer to have one trigger cloudflare worker to do tha...

conor•9/14/24, 6:34 PM

What events can you trigger on in D1 and to what resolution?

Can I trigger on row creation that meets specific criteria? E.g. a row was added with this value for that column?

yeeet•9/14/24, 10:12 PM

hi is it possible to save a d1 prepared statement in a global variable or something when my worker is initializing, so I can reuse it across multiple invocations? currently, I've got the following:

export class SampleEndpoint extends OpenAPIRoute {
  
  async handle(c) {
    const stmt = await c.env.DB.prepare(...);
  }
}

export class SampleEndpoint extends OpenAPIRoute {
  
  async handle(c) {
    const stmt = await c.env.DB.prepare(...);
  }
}

so unless there's some sort of caching under the hood, I guess it'd have to parse the query every time my handler gets called, which I want to avoid. Mainly, I'm not sure how to access the c.env.DBc.env.DB object from outside the handlehandle function of an endpoint

edit: actually i'm gonna introduce a db abstraction layer which caches prepared stmts in global variables and if there's no prepared stmt cached then it creates a fresh one. from reading online, it seems like global variables' consistency guarantees would b fine for this sort of use-case

o7•9/15/24, 12:01 AM

I've encountered a use case for dynamic bindings. Can't wait!

Ooof2win2 i am executing the queries in a new fetch request every time. i literally can’t ...

akashra•9/15/24, 8:15 AM

Right, so in this case, you should probably consider writing something that either enabled these to be batched from whatever is calling to the worker, or having them push to a queue which can periodically process data off that queue if you don't require synchronous responses.

oof2win2OP•9/15/24, 8:16 AM

sqlite should 100% be able to handle 300 writes per second

Ooof2win2 sqlite should 100% be able to handle 300 writes per second

akashra•9/15/24, 8:17 AM

It can, but I was more getting at why if you have a single process responsible for initiating all of these, why you're calling 300 separate queries when they don't need to be - or can be procesed without the overhead of locks in different sessions.

oof2win2OP•9/15/24, 8:18 AM

wym they don't need to be

oof2win2OP•9/15/24, 8:18 AM

its 300 separate http requests so 300 separate inserts make sense

akashra•9/15/24, 8:18 AM

you gotta remember how each worker is loading that data, obtaining the lock to write that data, then having to process that to write back and return.

akashra•9/15/24, 8:18 AM

yes - why are you doing 300 separate http requests - from the way this is a specific number, it sounds like they all come from the same client?

oof2win2OP•9/15/24, 8:19 AM

i just did 300 at once

oof2win2OP•9/15/24, 8:19 AM

they come from db triggers

oof2win2OP•9/15/24, 8:19 AM

it's possible i get only 1 and it's possible i get 2000

akashra•9/15/24, 8:20 AM

you're calling out so a db trigger fires a call to a http endpoint having access to the same db the event triggering it originated from?

i'm executing ~300 requests in parallel to simply run this (300 requests to the worker, each worker

Similar Threads

Similar Threads

Similar Threads