by "their own worker projects" do you mean making envs per PR / per person? that seems like it would

by "their own worker projects" do you mean making envs per PR / per person? that seems like it would blow up your workers dashboard.

i'm also curious if there are best practices here or if this is what cloudflare does internally?

Ssathoro probably cheaper and more efficient to just auto delete after some long period l...

lambrospetrou•2/10/25, 12:28 PM

Yeah, in this case you are literally talking about ephemeral data. So, setting a TTL for the alarm long enough ahead in time to do the cleanup is the right thing to do. How short or how long you do that it depends on the strictness of your application.

Kkchro3 by "their own worker projects" do you mean making envs per PR / per person? that...

lambrospetrou•2/10/25, 12:29 PM

Well if it's tens of extra projects, it's not a problem. If it's going to be hundreds of thousands then yeah it won't be nice experience. We use separate projects for each of our envs in some teams. Keep it simple.

Llambrospetrou Well if it's tens of extra projects, it's not a problem. If it's going to be hun...

kchro3OP•2/10/25, 12:33 PM

Do you run into the issue I was describing then where one person would create a DO and then everyone has to pull in those changes in order to deploy?

Kkchro3 Do you run into the issue I was describing then where one person would create a ...

lambrospetrou•2/10/25, 12:42 PM

No, although I am not sure I understood your full issue. In our case we don't deploy per PR. We only deploy after PRs are merged to a staging env, and later in production. So we don't have multiple PRs modifying the same workers.

Llambrospetrou No, although I am not sure I understood your full issue. In our case we don't de...

kchro3OP•2/10/25, 12:45 PM

ack, I think I might end up doing that too then. We do have a dev vs prod branch, but all of the PRs are deployed to a shared env

Kkchro3 ack, I think I might end up doing that too then. We do have a dev vs prod branch...

lambrospetrou•2/10/25, 12:46 PM

Also, if you are worried about too many script projects, you can extend your CI to delete the deployed workers/DOs once a PR is merged and closed. So you will only have as many projects as your open PRs.
Just something to think about too, if you like having the whole infra per PR.

Llambrospetrou Also, if you are worried about too many script projects, you can extend your CI ...

kchro3OP•2/10/25, 12:50 PM

ok, thanks. While I have you, I had a couple more questions:

1. for migrations, are you supposed to just have a running changelog of all tags? Or do you delete them after there are no more references to deleted/renamed classes?
2. Do wrangler.toml files support dynamic envs? Like if I wanted a user specific env or a PR specific env, can I have like [[env.$USER.durable_objects]] or something like that?

Kkchro3 ok, thanks. While I have you, I had a couple more questions: 1. for migrations,...

lambrospetrou•2/10/25, 12:51 PM

1. As far as I know yes. You should keep migrations there. I think of them as immutable once deployed.
2. Yes, absolutely. Example: https://github.com/lambrospetrou/rediflare/blob/b2784e0674474d4d1372d89a74be1d3ed9cc63ad/wrangler.toml#L55

Llambrospetrou 1. As far as I know yes. You should keep migrations there. I think of them as im...

kchro3OP•2/10/25, 12:52 PM

great, thanks so much

Llambrospetrou 1. As far as I know yes. You should keep migrations there. I think of them as im...

kchro3OP•2/10/25, 12:53 PM

Ah wait, but in the example you shared, the env is hardcoded as staging right?

Kkchro3 Ah wait, but in the example you shared, the env is hardcoded as staging right?

lambrospetrou•2/10/25, 12:55 PM

Yes, but you can generate your own wrangler configuration right before deploying, depending on your CI.
For example, you could have a

wrangler.toml/json

wrangler.toml/json

that has

[env.__REPLACEME__.durable_objects]

[env.__REPLACEME__.durable_objects]

, and you replace that placeholder in your CI with the PR identifier (ie..

PR_ID

PR_ID

), and then run

wrangler deploy -c path/to/wrangler.toml --env=<PR_ID>

wrangler deploy -c path/to/wrangler.toml --env=<PR_ID>

Llambrospetrou Yes, but you can generate your own wrangler configuration right before deploying...

kchro3OP•2/10/25, 12:57 PM

I see, thanks

Burrito•2/10/25, 8:37 PM

Related, since R2

put

put

accepts a

ReadableStream

ReadableStream

, I was also wonder that if it's possible to implement streaming exporting a DO's storage this way.

Nnil I'm returning a readable stream inside of a response from a durable object. If I...

jed•2/10/25, 8:57 PM

can you use https://developer.mozilla.org/en-US/docs/Web/API/TextEncoderStream ?

MDN Web Docs

TextEncoderStream - Web APIs | MDN

The TextEncoderStream interface of the Encoding API converts a stream of strings into bytes in the UTF-8 encoding. It is the streaming equivalent of TextEncoder.

Llambrospetrou Speculating here since I didn't run your code, but the `env` you get in the DO c...

DiamondDragon•2/10/25, 9:01 PM

Thanks - what I needed to do was pass

Env[Bindings]

Env[Bindings]

to the DO.

else•2/10/25, 9:19 PM

are there plans to expand DO availability to other regions?

else•2/10/25, 9:19 PM

for example, currently there are no DOs spawned near india

Eelse for example, currently there are no DOs spawned near india

1984 Ford Laser•2/10/25, 10:23 PM

India and CF don't exactly get along well

Ben•2/10/25, 10:41 PM

Question, is there an easy way to arbitrary query a durable object sqlite database but restrict it to reads?

For example I need to query raw sql to debug issues in prod, but I don't want it to be easy to be able to accidentally drop rows and tables etc

1984 Ford Laser•2/11/25, 7:28 AM

Does JSON querying work in DO SQLite like it does in D1? What's the main differences between the two implementations that I need to be aware of

11984 Ford Laser Does JSON querying work in DO SQLite like it does in D1? What's the main differe...

lambrospetrou•2/11/25, 10:29 AM

Same.

1984 Ford Laser•2/11/25, 10:32 AM

Do SQL triggers work too?

Avi•2/11/25, 8:21 PM

is there a way to configure my DO/worker to NOT spawn in particular regions?

Avi•2/11/25, 8:22 PM

i'm running into an issue where users in Hong Kong are getting their DO created in that region, but that means OpenAI API calls

AAvi i'm running into an issue where users in Hong Kong are getting their DO created ...

Chaika•2/11/25, 9:08 PM

hmm that's interesting, maybe just checking if their region is Asia Pacific (and/or colo, but colos can change), and then using a hint to oceanic region rather then apac? APAC has Hong kong as part of it, so have to avoid it entirely

Advany•2/11/25, 9:29 PM

https://x.com/chronark_/status/1889411951601783129

We are using do's a lot and slowly scaling up. 500 rps doesnt seem like a lot.

Is this the maximum rate limit of DO's?

chronark — oss/acc (@chronark_) on X

@advany @MrAhmadAwais @Cloudflare @PlanetScale @DrizzleORM it just adds up

worker invocation: $0.30/mil
do invocation: $0.15/mil
do cputime: hard to predict
do storage read: $0.20/4kb/mil

we also observe durable objects randomly failing at 500+ RPS (on a single DO) with "internal error" or "network error", which isn't super helpful to

•

2/11/25, 8:31 PM

CChaika hmm that's interesting, maybe just checking if their region is Asia Pacific (and...

Avi•2/11/25, 9:32 PM

idea being that I would look at the CF headers on the initial request and do something like:

if location is hong -> set

locationHint

locationHint

"Oceania"

"Oceania"

AAdvany https://x.com/chronark_/status/1889411951601783129 We are using do's a lot and ...

1984 Ford Laser•2/11/25, 9:36 PM

Then you aren't using it correctly. The RPS will depend on what work you are doing with each DO. You should be scaling up to many instances if you are overloading, your architecture isn't designed for your use case

11984 Ford Laser Then you aren't using it correctly. The RPS will depend on what work you are doi...

Advany•2/11/25, 10:00 PM

So many read instances and only one write instance right?

1984 Ford Laser•2/11/25, 10:02 PM

Not necessarily, DO instances are single threaded. How you use them will depend on what work you are doing, and how the work needs to be distributed. If you are erroring out at 500RPS you are overloading them and should either shard the work out to many instances, or split your architecture in different ways. There are many ways to use DOs, hitting RPS limits is a failure of implementation, not of the platform

1984 Ford Laser•2/11/25, 10:06 PM

I think you misunderstand how DOs work a bit,. Each instance is isolated from one another, so having different read instances wouldn't work unless they are reading from a central DB versus the storage attached to the instance

Ggnomesgames If that helps as an example, for my theatre tickets website, I use DO to store t...

1984 Ford Laser•2/11/25, 11:08 PM

Example is good. I have a DO per year, a DO per sports league, a DO per sports match, a DO per live stream for each match

1984 Ford Laser•2/11/25, 11:09 PM

Different clients access different DOs depending on usage, and other Workers and DOs access these ones as needed

Ggnomesgames If that helps as an example, for my theatre tickets website, I use DO to store t...

lambrospetrou•2/11/25, 11:24 PM

Obviously it depends on your traffic, but this is a classic example of how hot partitions can bring your site down

With a single DO handling the whole show, if you launch tickets for a big show and you get thousands of visitors, then that's not going to work very well.
If the traffic is low or well spread out, then this is very nice and works well, though.

So, tradeoffs.

Mozzy•2/12/25, 12:07 AM

Can I have a DO backed by both SQL and KV?
I want to use KV storage for some data and SQL for other data

MMozzy Can I have a DO backed by both SQL and KV? I want to use KV storage for some dat...

Chaika•2/12/25, 12:11 AM

No, but if you use SQL, the KV APIs still work, same api, and just use a hidden table

DanTheGoodman•2/12/25, 12:13 AM

Can you make it such that tests wipe previous DO storage data locally when starting the test? Or delete on exit?

CChaika No, but if you use SQL, the KV APIs still work, same api, and just use a hidden ...

Mozzy•2/12/25, 12:13 AM

neat

DDanTheGoodman Can you make it such that tests wipe previous DO storage data locally when start...

Burrito•2/12/25, 1:24 AM

If you are using #vitest-integration-beta, there's an

isolatedStorage

isolatedStorage

option, which IIRC is on by default.

BBurrito If you are using #vitest-integration-beta, there's an `isolatedStorage` option, ...

DanTheGoodman•2/12/25, 1:34 AM

maybe not the beta, im def runnign into issues. Also where does it save to so i can delete it?

Burrito•2/12/25, 1:34 AM

Well, that answer is assuming you are using Vitest integration. If you are using something else then I'm not sure.

Llambrospetrou Obviously it depends on your traffic, but this is a classic example of how hot p...

makrdev•2/12/25, 1:05 PM

I was planning to use a single DO per slug to route traffic to different variations but after I saw your comment I’ve realized that its not a good pattern.

How can we architect such a system like this with DOs then?

Mmakrdev I was planning to use a single DO per slug to route traffic to different variati...

lambrospetrou•2/12/25, 1:10 PM

As always it depends on volume. If you have a few hundred requests per second then it should be fine either way.
Once you have thousands per second per DO it's where you need to start sharding more. There is no "general pattern" that works for everywhere. Different scales need different solutions. And obviously complexity scales as well.
I just mentioned that case because for example if something like Prime Video designed 1 DO per show, then at that scale it wouldn't work. If a local radio station did it, probably all is fine.

by "their own worker projects" do you mean making envs per PR / per person? that seems like it would

Similar Threads