I still don't follow the use-case. You have say 1000 DOs with storage used for actual data. But, now

I still don't follow the use-case. You have say 1000 DOs with storage used for actual data. But, now you deleted the entities related to those DOs from a different part of your applications but forgot about the DOs, and you are wondering how to delete those 1000 DOs?

To phrase it differently, what do you need the DOs to do in this example you have in mind?

Llambrospetrou I still don't follow the use-case. You have say 1000 DOs with storage used for a...

Hard@Work•2/10/25, 12:02 PM

Yeah. Like, I have an app that generates a random value, stores it as a cookie in the user's browser, and then that random value is used as the Name for a DO ID. However, I neglected to store the random value in a separate datastore.

HHard@Work Yeah. Like, I have an app that generates a random value, stores it as a cookie i...

lambrospetrouOP•2/10/25, 12:03 PM

But that's like saying you stored an object in R2 with a random object key, and now you lost that key. You need to somehow iterate the bucket and find which ones are orphan and delete them. Same thing with DOs.

Llambrospetrou But that's like saying you stored an object in R2 with a random object key, and ...

Hard@Work•2/10/25, 12:04 PM

Yeah, I was just wondering because it seems like you advised against using the List API. At least that was the vibe I was getting, let me know if I am wrong

HHard@Work Yeah, I was just wondering because it seems like you advised against using the L...

lambrospetrouOP•2/10/25, 12:06 PM

Hmm, I didn't advise against using the list API. I advised against using it as part of your hot path in your application logic. If your application is designed in a way that you have to iterate all DOs in a namespace, or thousands of them, and you are not necessarily accessing their data to serve that query, then the architecture of the application is wrong.

Llambrospetrou Hmm, I didn't advise against using the list API. I advised against using it as p...

Hard@Work•2/10/25, 12:08 PM

Ok, that makes sense. I was also just trying to put myself in the shoes of someone who may not have familiarity with the platform, or that they should be architecting their app for autocleanup/storing IDs somewhere.

HHard@Work Ok, that makes sense. I was also just trying to put myself in the shoes of someo...

lambrospetrouOP•2/10/25, 12:14 PM

that they should be architecting their app for autocleanup/storing IDs somewhere

I still don't agree with this. Autocleaning are very different things from storing IDs.

You almost always want to have a way to get the DO ID from within your application. You cannot always trust users telling you which DO ID they have access to, you need to serve List APIs, and hundreds other use cases. Even if don't store literally the DO IDs, you should have a deterministic way of generating that ID. Otherwise it's a black hole. Like any other storage product, when an entity/account/resource will be deleted you need to do cleanup. Same applies for R2, for DOs, for SQL databases when the IDs are random, and the list goes on.

Autocleaning, not sure where this fits. Maybe just for temporary ephemeral data you want to delete after a TTL, but again, there shouldn't be magic, you design around this deletion.

Maybe it's just me, but I don't want providers to be deleting my data on their own or through something I didn't explicitly configured/coded.

Llambrospetrou > that they should be architecting their app for autocleanup/storing IDs somewhe...

Hard@Work•2/10/25, 12:19 PM

By Autocleaning, I mean like scheduling an alarm that deletes everything stored, if it hasn't been renewed before then.

As an example, in a few of my apps, I use a DO as a more secure cookie. I.e., while the client just sees an opaque value stored, I can ask the DO corresponding to that opaque value, and it returns a user id, their expected user agent/location, etc.. However, because cookies themselves don't persist forever, I want to ensure that the DOs don't stick around forever either, as that would cost me money for no benefit. Thus, I set an alarm for the time that the cookie would expire, and then on every call to the DO, I reset the alarm. If the alarm fires, it deletes all of its storage, so that I'm no longer billed for it

Hard@Work•2/10/25, 12:20 PM

I don't know if this is the way

you are supposed to go about this kind of work, but it seems to work ok for now

Hard@Work•2/10/25, 12:21 PM

For the above case, while I could store every ID I've ever used in a second database somewhere, if I do that, I might as well just not use a DO, and store the metadata in that database.

HHard@Work By Autocleaning, I mean like scheduling an alarm that deletes everything stored,...

sathoro•2/10/25, 12:24 PM

probably cheaper and more efficient to just auto delete after some long period like a few months rather than writing to the alarm every single request

sathoro•2/10/25, 12:25 PM

its not like you are storing a lot of data

Llambrospetrou If you want folks to be able to add/remove DOs on their own without impacting th...

kchro3•2/10/25, 12:25 PM

by "their own worker projects" do you mean making envs per PR / per person? that seems like it would blow up your workers dashboard.

i'm also curious if there are best practices here or if this is what cloudflare does internally?

Ssathoro probably cheaper and more efficient to just auto delete after some long period l...

lambrospetrouOP•2/10/25, 12:28 PM

Yeah, in this case you are literally talking about ephemeral data. So, setting a TTL for the alarm long enough ahead in time to do the cleanup is the right thing to do. How short or how long you do that it depends on the strictness of your application.

Kkchro3 by "their own worker projects" do you mean making envs per PR / per person? that...

lambrospetrouOP•2/10/25, 12:29 PM

Well if it's tens of extra projects, it's not a problem. If it's going to be hundreds of thousands then yeah it won't be nice experience. We use separate projects for each of our envs in some teams. Keep it simple.

Llambrospetrou Well if it's tens of extra projects, it's not a problem. If it's going to be hun...

kchro3•2/10/25, 12:33 PM

Do you run into the issue I was describing then where one person would create a DO and then everyone has to pull in those changes in order to deploy?

Kkchro3 Do you run into the issue I was describing then where one person would create a ...

lambrospetrouOP•2/10/25, 12:42 PM

No, although I am not sure I understood your full issue. In our case we don't deploy per PR. We only deploy after PRs are merged to a staging env, and later in production. So we don't have multiple PRs modifying the same workers.

Llambrospetrou No, although I am not sure I understood your full issue. In our case we don't de...

kchro3•2/10/25, 12:45 PM

ack, I think I might end up doing that too then. We do have a dev vs prod branch, but all of the PRs are deployed to a shared env

Kkchro3 ack, I think I might end up doing that too then. We do have a dev vs prod branch...

lambrospetrouOP•2/10/25, 12:46 PM

Also, if you are worried about too many script projects, you can extend your CI to delete the deployed workers/DOs once a PR is merged and closed. So you will only have as many projects as your open PRs.
Just something to think about too, if you like having the whole infra per PR.

Llambrospetrou Also, if you are worried about too many script projects, you can extend your CI ...

kchro3•2/10/25, 12:50 PM

ok, thanks. While I have you, I had a couple more questions:

1. for migrations, are you supposed to just have a running changelog of all tags? Or do you delete them after there are no more references to deleted/renamed classes?
2. Do wrangler.toml files support dynamic envs? Like if I wanted a user specific env or a PR specific env, can I have like [[env.$USER.durable_objects]] or something like that?

Kkchro3 ok, thanks. While I have you, I had a couple more questions: 1. for migrations,...

lambrospetrouOP•2/10/25, 12:51 PM

1. As far as I know yes. You should keep migrations there. I think of them as immutable once deployed.
2. Yes, absolutely. Example: https://github.com/lambrospetrou/rediflare/blob/b2784e0674474d4d1372d89a74be1d3ed9cc63ad/wrangler.toml#L55

Llambrospetrou 1. As far as I know yes. You should keep migrations there. I think of them as im...

kchro3•2/10/25, 12:52 PM

great, thanks so much

Llambrospetrou 1. As far as I know yes. You should keep migrations there. I think of them as im...

kchro3•2/10/25, 12:53 PM

Ah wait, but in the example you shared, the env is hardcoded as staging right?

Kkchro3 Ah wait, but in the example you shared, the env is hardcoded as staging right?

lambrospetrouOP•2/10/25, 12:55 PM

Yes, but you can generate your own wrangler configuration right before deploying, depending on your CI.
For example, you could have a

wrangler.toml/json

wrangler.toml/json

that has

[env.__REPLACEME__.durable_objects]

[env.__REPLACEME__.durable_objects]

, and you replace that placeholder in your CI with the PR identifier (ie..

PR_ID

PR_ID

), and then run

wrangler deploy -c path/to/wrangler.toml --env=<PR_ID>

wrangler deploy -c path/to/wrangler.toml --env=<PR_ID>

Llambrospetrou Yes, but you can generate your own wrangler configuration right before deploying...

kchro3•2/10/25, 12:57 PM

I see, thanks

Burrito•2/10/25, 8:37 PM

Related, since R2

put

put

accepts a

ReadableStream

ReadableStream

, I was also wonder that if it's possible to implement streaming exporting a DO's storage this way.

Nnil I'm returning a readable stream inside of a response from a durable object. If I...

jed•2/10/25, 8:57 PM

can you use https://developer.mozilla.org/en-US/docs/Web/API/TextEncoderStream ?

MDN Web Docs

TextEncoderStream - Web APIs | MDN

The TextEncoderStream interface of the Encoding API converts a stream of strings into bytes in the UTF-8 encoding. It is the streaming equivalent of TextEncoder.

Llambrospetrou Speculating here since I didn't run your code, but the `env` you get in the DO c...

DiamondDragon•2/10/25, 9:01 PM

Thanks - what I needed to do was pass

Env[Bindings]

Env[Bindings]

to the DO.

else•2/10/25, 9:19 PM

are there plans to expand DO availability to other regions?

else•2/10/25, 9:19 PM

for example, currently there are no DOs spawned near india

Eelse for example, currently there are no DOs spawned near india

1984 Ford Laser•2/10/25, 10:23 PM

India and CF don't exactly get along well

Ben•2/10/25, 10:41 PM

Question, is there an easy way to arbitrary query a durable object sqlite database but restrict it to reads?

For example I need to query raw sql to debug issues in prod, but I don't want it to be easy to be able to accidentally drop rows and tables etc

1984 Ford Laser•2/11/25, 7:28 AM

Does JSON querying work in DO SQLite like it does in D1? What's the main differences between the two implementations that I need to be aware of

11984 Ford Laser Does JSON querying work in DO SQLite like it does in D1? What's the main differe...

lambrospetrouOP•2/11/25, 10:29 AM

Same.

1984 Ford Laser•2/11/25, 10:32 AM

Do SQL triggers work too?

Avi•2/11/25, 8:21 PM

is there a way to configure my DO/worker to NOT spawn in particular regions?

Avi•2/11/25, 8:22 PM

i'm running into an issue where users in Hong Kong are getting their DO created in that region, but that means OpenAI API calls

AAvi i'm running into an issue where users in Hong Kong are getting their DO created ...

Chaika•2/11/25, 9:08 PM

hmm that's interesting, maybe just checking if their region is Asia Pacific (and/or colo, but colos can change), and then using a hint to oceanic region rather then apac? APAC has Hong kong as part of it, so have to avoid it entirely

Advany•2/11/25, 9:29 PM

https://x.com/chronark_/status/1889411951601783129

We are using do's a lot and slowly scaling up. 500 rps doesnt seem like a lot.

Is this the maximum rate limit of DO's?

chronark — oss/acc (@chronark_) on X

@advany @MrAhmadAwais @Cloudflare @PlanetScale @DrizzleORM it just adds up

worker invocation: $0.30/mil
do invocation: $0.15/mil
do cputime: hard to predict
do storage read: $0.20/4kb/mil

we also observe durable objects randomly failing at 500+ RPS (on a single DO) with "internal error" or "network error", which isn't super helpful to

•

2/11/25, 8:31 PM

CChaika hmm that's interesting, maybe just checking if their region is Asia Pacific (and...

Avi•2/11/25, 9:32 PM

idea being that I would look at the CF headers on the initial request and do something like:

if location is hong -> set locationHintlocationHint to "Oceania""Oceania"?

AAdvany https://x.com/chronark_/status/1889411951601783129 We are using do's a lot and ...

1984 Ford Laser•2/11/25, 9:36 PM

Then you aren't using it correctly. The RPS will depend on what work you are doing with each DO. You should be scaling up to many instances if you are overloading, your architecture isn't designed for your use case

11984 Ford Laser Then you aren't using it correctly. The RPS will depend on what work you are doi...

Advany•2/11/25, 10:00 PM

So many read instances and only one write instance right?

1984 Ford Laser•2/11/25, 10:02 PM

Not necessarily, DO instances are single threaded. How you use them will depend on what work you are doing, and how the work needs to be distributed. If you are erroring out at 500RPS you are overloading them and should either shard the work out to many instances, or split your architecture in different ways. There are many ways to use DOs, hitting RPS limits is a failure of implementation, not of the platform

1984 Ford Laser•2/11/25, 10:06 PM

I think you misunderstand how DOs work a bit,. Each instance is isolated from one another, so having different read instances wouldn't work unless they are reading from a central DB versus the storage attached to the instance

Ggnomesgames If that helps as an example, for my theatre tickets website, I use DO to store t...

1984 Ford Laser•2/11/25, 11:08 PM

Example is good. I have a DO per year, a DO per sports league, a DO per sports match, a DO per live stream for each match

I still don't follow the use-case. You have say 1000 DOs with storage used for actual data. But, now

Similar Threads