Cloudflare Developers•16mo ago

Similar to the above, if I'm running to workers locally, how can I get them to share the same databa

Similar to the above, if I'm running to workers locally, how can I get them to share the same database?

shixzie•9/14/24, 12:15 AM

are there any d1-related anouncements planned for birthday week?

Ooof2win2 i'm executing ~300 requests in parallel to simply run this (300 requests to the ...

akashra•9/14/24, 4:20 AM

Is there some particular reason you're not using a single prepared statement with multiple param sets called as a single query?

Aakashra Is there some particular reason you're not using a single prepared statement wit...

oof2win2•9/14/24, 6:19 AM

i am executing the queries in a new fetch request every time. i literally can’t have multiple params as the requests are independent of each other

Hello, I’m Allie!•9/14/24, 8:58 AM

Point them to the same directory with —persist-to—persist-to

Nnulled Do you mean an UPDATE trigger that updates a timestamp on a table each time you ...

acan•9/14/24, 1:38 PM

I am surprised that D1 supports triggers. It's great.

Aacan I am surprised that D1 supports triggers. It's great.

Puliczek•9/14/24, 5:16 PM

I wish there could be cloudflare worker trigger on D1 table new row.

Puliczek•9/14/24, 5:17 PM

I would like to create new payment/new user -> send notification to telegram/discord.

HHello, I’m Allie!Point them to the same directory with `—persist-to`

conorOP•9/14/24, 6:09 PM

I tried that, state persists but it doesn't have my migrations applied and doesn't seem to want to apply them (no migrations to apply!). ~~Do I have to use the same flag for the migrations command I wonder?~~

Same flag not applicable to the migrations command and I'm not sure there's an equivalent.

I get:

✘ [ERROR] Implications failed:

   persist-to -> local

✘ [ERROR] Implications failed:

   persist-to -> local

But when I use --local--local instead of persist-topersist-to for migrations I get :white_check_mark: No migrations to apply!:white_check_mark: No migrations to apply!

conorOP•9/14/24, 6:12 PM

Running wrangler 3.78.0wrangler 3.78.0

conorOP•9/14/24, 6:21 PM

Ah ok! You need to supply both the --persist-to--persist-to flag and the --local--local flag. I imagined --persist-to--persist-to would imply locallocal because I didn't think it was possible to direct persistence to a path remotely?

Original message was deleted

Puliczek•9/14/24, 6:33 PM

I have 10+ db and apps, I prefer to have one trigger cloudflare worker to do that, without CRON. Or maybe CRON 1hr should be ok, maybe thats the way.

PPuliczek I have 10+ db and apps, I prefer to have one trigger cloudflare worker to do tha...

conorOP•9/14/24, 6:34 PM

What events can you trigger on in D1 and to what resolution?

Can I trigger on row creation that meets specific criteria? E.g. a row was added with this value for that column?

yeeet•9/14/24, 10:12 PM

hi is it possible to save a d1 prepared statement in a global variable or something when my worker is initializing, so I can reuse it across multiple invocations? currently, I've got the following:

export class SampleEndpoint extends OpenAPIRoute {
  
  async handle(c) {
    const stmt = await c.env.DB.prepare(...);
  }
}

export class SampleEndpoint extends OpenAPIRoute {
  
  async handle(c) {
    const stmt = await c.env.DB.prepare(...);
  }
}

so unless there's some sort of caching under the hood, I guess it'd have to parse the query every time my handler gets called, which I want to avoid. Mainly, I'm not sure how to access the c.env.DBc.env.DB object from outside the handlehandle function of an endpoint

edit: actually i'm gonna introduce a db abstraction layer which caches prepared stmts in global variables and if there's no prepared stmt cached then it creates a fresh one. from reading online, it seems like global variables' consistency guarantees would b fine for this sort of use-case

o7•9/15/24, 12:01 AM

I've encountered a use case for dynamic bindings. Can't wait!

Ooof2win2 i am executing the queries in a new fetch request every time. i literally can’t ...

akashra•9/15/24, 8:15 AM

Right, so in this case, you should probably consider writing something that either enabled these to be batched from whatever is calling to the worker, or having them push to a queue which can periodically process data off that queue if you don't require synchronous responses.

oof2win2•9/15/24, 8:16 AM

sqlite should 100% be able to handle 300 writes per second

Ooof2win2 sqlite should 100% be able to handle 300 writes per second

akashra•9/15/24, 8:17 AM

It can, but I was more getting at why if you have a single process responsible for initiating all of these, why you're calling 300 separate queries when they don't need to be - or can be procesed without the overhead of locks in different sessions.

oof2win2•9/15/24, 8:18 AM

wym they don't need to be

oof2win2•9/15/24, 8:18 AM

its 300 separate http requests so 300 separate inserts make sense

akashra•9/15/24, 8:18 AM

you gotta remember how each worker is loading that data, obtaining the lock to write that data, then having to process that to write back and return.

akashra•9/15/24, 8:18 AM

yes - why are you doing 300 separate http requests - from the way this is a specific number, it sounds like they all come from the same client?

oof2win2•9/15/24, 8:19 AM

i just did 300 at once

oof2win2•9/15/24, 8:19 AM

they come from db triggers

oof2win2•9/15/24, 8:19 AM

it's possible i get only 1 and it's possible i get 2000

akashra•9/15/24, 8:20 AM

you're calling out so a db trigger fires a call to a http endpoint having access to the same db the event triggering it originated from?

akashra•9/15/24, 8:21 AM

I'm trying to figure out how you've ended up in a situation where you have hundreds of queries needing to be executed as the result of any single event, such that they can't be batched - unless they're a synchronous chain

Aakashra you're calling out so a db trigger fires a call to a http endpoint having access...

oof2win2•9/15/24, 8:22 AM

it doesnt have access to the same db

oof2win2•9/15/24, 8:22 AM

it's from an external supabase db and here i am doing external processing + batching

akashra•9/15/24, 8:23 AM

okay, but what's the process flow look like such that those calls can't be batched to justify writing a single service endpoint if you have a situation that necessitates making a large number of calls to the d1 db (which you're currently doing via a worker for each?)

oof2win2•9/15/24, 8:25 AM

i thought the point of workers+d1 was that it could handle this usecase, adding a whole queue seems like overkill to me

akashra•9/15/24, 8:26 AM

the performance hit you're likely complaining about here is not that sqlite can't handle 300 (or N) queries in quick succession. It's that you have potentially that many separate processes trying to obtain a connection and lock on that db which have to be handled in a queue.

oof2win2•9/15/24, 8:27 AM

yes i get that, but i thought that was the point of cf workers - they could just handle that

oof2win2•9/15/24, 8:27 AM

like if you have a realworld application with hundreds of requests per second it should "just work"

akashra•9/15/24, 8:28 AM

for read-heavy instances on the same data, sure.

oof2win2•9/15/24, 8:30 AM

well in that case something write heavy would make more sense here

oof2win2•9/15/24, 8:31 AM

ah well

akashra•9/15/24, 8:35 AM

I'm trying to see if I can find the tech architecture article on how workers and D1 DBs are exposed to a worker and kept in sync - that might make it make more sense as to why you would run in to issues with what you're trying to do here.

So yes, from the limited detail you've given, it sounds more like an appropriate use-case for a more persistent DB.

Yyeeet hi is it possible to save a d1 prepared statement in a global variable or someth...

kian•9/15/24, 12:24 PM

This isn’t necessary, prepared statements in your Worker are just a JS class.

kian•9/15/24, 12:24 PM

(calling prepare never talks to your DB yet, only when you run it)

kian•9/15/24, 12:25 PM

All of the statement reuse is handled by D1 itself.

Ephemeros•9/15/24, 12:51 PM

what's the standard way of initializing a database's values? would that be in a migration sql?

Ephemeros•9/15/24, 12:51 PM

like, i want to seed some default admin user etc

Deep•9/15/24, 2:40 PM

hi is there anyway to sync a local sqlite db for a desktop app with D1?

Kkian This isn’t necessary, prepared statements in your Worker are just a JS class.

yeeet•9/15/24, 2:54 PM

Thanks

cosbgn•9/15/24, 4:55 PM

Has cloudflare ever thought about allowing https://github.com/asg017/sqlite-vec on D1?

I know about vectorize, but having everything as simple as SQLITE would be unbeatable. I know turso does it and it seems to work very well.

cosbgn•9/15/24, 4:56 PM

it would allow for much better metadata filtering etc having the whole power of SQL

cosbgn•9/15/24, 5:05 PM

Can I get it to work simply using: npm:sqlite-vecnpm:sqlite-vec?
Like on this: https://alexgarcia.xyz/sqlite-vec/js.html

Ccosbgn Can I get it to work simply using: `npm:sqlite-vec`? Like on this: https://alexg...

kian•9/15/24, 5:20 PM

On Workers? No, you won't be able to use that package.

Vijin•9/16/24, 5:29 AM

Hi, i am new to D1, i would like to use D1 for my production application. Can you guys please give a clarification on the below doubts
1. Is D1 Production Ready?
2. Saw that D1 has a maximum size limit of 10GB per Database in paid plan. Can we upgrade that?

Dinfer•9/16/24, 8:17 AM

Is transaction supported in future?

Similar to the above, if I'm running to workers locally, how can I get them to share the same databa

Similar Threads

Similar Threads

Similar Threads