I’m wondering though, for cases where you store a little bit of data but then forget to clean it up,

I’m wondering though, for cases where you store a little bit of data but then forget to clean it up, or do have machinery to clean up that just happened not to work in that case, what should you do? A few bytes here and there won’t do much, but it could become a problem at some point, depending on scale.

One fix is always to nuke the entire namespace, but that sucks if you have data you do want to keep

HHello, I’m Allie!I’m wondering though, for cases where you store a little bit of data but then fo...

lambrospetrou•2/10/25, 12:28 AM

What does "forget" mean though? I cannot give an answer as generic as that. If a resource has been deleted from the rest of your application and not the DO storage, then you should have a trail of the deletion somewhere in the system, and maybe even do a crawl after the fact to clean up or something.

kchro3•2/10/25, 1:03 AM

I have a usability question with Durable Objects. We have a CICD pipeline where we stage our branches to a stage env, but because of how DOs do migrations, if one person adds a DO in a PR, everyone else’s PR start failing to deploy because it’s missing a delete tag for a Durable Object that doesn’t exist yet.

Is there a recommended way to do CICD here?

11984 Ford Laser Do SQL triggers work in DOs? I swear someone posted that they'd gotten it workin...

1984 Ford Laser•2/10/25, 7:28 AM

Anyone have any ideas on this one?

DiamondDragon•2/10/25, 8:18 AM

can someone help me here? I have this general basic setup and am trying to do R2 operations within a DO but , but i cannot figure out why the env bindings are not being recognized . I get

[ERROR] TypeError: Cannot read properties of undefined (reading 'MY_BUCKET')

[ERROR] TypeError: Cannot read properties of undefined (reading 'MY_BUCKET')

. I can do R2 operations in the hono routes but not within the DO at the moment using the bindings

The use case is i need to upload a large CSV file so need to use R2 storage rather than DO storage

I am running this local.

type Bindings = {
    MY_BUCKET: R2Bucket
    MY_DO: DurableObjectNamespace<MyDurableObject>
}

type Variables = {}

export type Env = {
    Bindings: Bindings
    Variables: Variables
}

type Bindings = {
    MY_BUCKET: R2Bucket
    MY_DO: DurableObjectNamespace<MyDurableObject>
}

type Variables = {}

export type Env = {
    Bindings: Bindings
    Variables: Variables
}

Durable Object Definition

export class MyDurableObject extends DurableObject<Env> {
    storage: DurableObjectStorage

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)
        this.storage = ctx.storage
        console.log('DO constructor', this.env)
    }

    async test() {
        const upload = await this.env.Bindings.MY_BUCKET.put(`test`, 'test')
        return upload
    }
}

export class MyDurableObject extends DurableObject<Env> {
    storage: DurableObjectStorage

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)
        this.storage = ctx.storage
        console.log('DO constructor', this.env)
    }

    async test() {
        const upload = await this.env.Bindings.MY_BUCKET.put(`test`, 'test')
        return upload
    }
}

Calling the Durable Object in Hono

// do imported above

const id = c.env.MY_DO.idFromName('my-durable-object')
const durableObject = c.env.MY_DO.get(id)
const zipFile = await durableObject.test()
return c.json({ message: 'Test', zipFile })


export {MY_DO}

// do imported above

const id = c.env.MY_DO.idFromName('my-durable-object')
const durableObject = c.env.MY_DO.get(id)
const zipFile = await durableObject.test()
return c.json({ message: 'Test', zipFile })


export {MY_DO}

Logging and Error Output

DO constructor {
    MY_DO: DurableObjectNamespace {},
    MY_BUCKET: R2Bucket {},

  }
  ✘ [ERROR] TypeError: Cannot read properties of undefined (reading 'MY_BUCKET')

DO constructor {
    MY_DO: DurableObjectNamespace {},
    MY_BUCKET: R2Bucket {},

  }
  ✘ [ERROR] TypeError: Cannot read properties of undefined (reading 'MY_BUCKET')

DDiamondDragon can someone help me here? I have this general basic setup and am trying to do R2...

lambrospetrou•2/10/25, 9:30 AM

Speculating here since I didn't run your code, but the

env

env

you get in the DO constructor has nothing to do with the Hono env. In your code it should be the

Bindings

Bindings

directly. So doing

await this.env.MY_BUCKET.put(...)

await this.env.MY_BUCKET.put(...)

will probably work.
And your logging BTW confirms this.
DOs constructor receive the Worker's definition of Env, as defined in your wrangler configuration.

Kkchro3 I have a usability question with Durable Objects. We have a CICD pipeline where ...

lambrospetrou•2/10/25, 9:33 AM

If you want folks to be able to add/remove DOs on their own without impacting the rest, then each PR or person should get their own worker projects so they will be different DO namespaces. This of course means someone's worker won't communicate with someone else's.

Llambrospetrou What does "forget" mean though? I cannot give an answer as generic as that. If a...

Hard@Work•2/10/25, 11:35 AM

Like, I have a few apps that are reasonably simple, a few lines of DO code, and a few i/o calls. While I know now to be more thorough, and ensure that the DOs clean themselves up, I didn't always, nor do I think that people who are starting fresh on the platform would know to do so

Hard@Work•2/10/25, 11:37 AM

While for production apps, there should always be a cleanup mechanism, I do think that it might be unreasonable to expect first-time Cloudflare/DevPlatform users to know they need a full DO ops logging system/AutoCleanup tool right off the bat. The "Delete the entire namespace" escape hatch is still there as a cleanup function, but it seems a tad heavyhanded

HHard@Work Like, I have a few apps that are reasonably simple, a few lines of DO code, and ...

lambrospetrou•2/10/25, 12:00 PM

I still don't follow the use-case. You have say 1000 DOs with storage used for actual data. But, now you deleted the entities related to those DOs from a different part of your applications but forgot about the DOs, and you are wondering how to delete those 1000 DOs?

To phrase it differently, what do you need the DOs to do in this example you have in mind?

Llambrospetrou I still don't follow the use-case. You have say 1000 DOs with storage used for a...

Hard@Work•2/10/25, 12:02 PM

Yeah. Like, I have an app that generates a random value, stores it as a cookie in the user's browser, and then that random value is used as the Name for a DO ID. However, I neglected to store the random value in a separate datastore.

HHard@Work Yeah. Like, I have an app that generates a random value, stores it as a cookie i...

lambrospetrou•2/10/25, 12:03 PM

But that's like saying you stored an object in R2 with a random object key, and now you lost that key. You need to somehow iterate the bucket and find which ones are orphan and delete them. Same thing with DOs.

Llambrospetrou But that's like saying you stored an object in R2 with a random object key, and ...

Hard@Work•2/10/25, 12:04 PM

Yeah, I was just wondering because it seems like you advised against using the List API. At least that was the vibe I was getting, let me know if I am wrong

HHard@Work Yeah, I was just wondering because it seems like you advised against using the L...

lambrospetrou•2/10/25, 12:06 PM

Hmm, I didn't advise against using the list API. I advised against using it as part of your hot path in your application logic. If your application is designed in a way that you have to iterate all DOs in a namespace, or thousands of them, and you are not necessarily accessing their data to serve that query, then the architecture of the application is wrong.

Llambrospetrou Hmm, I didn't advise against using the list API. I advised against using it as p...

Hard@Work•2/10/25, 12:08 PM

Ok, that makes sense. I was also just trying to put myself in the shoes of someone who may not have familiarity with the platform, or that they should be architecting their app for autocleanup/storing IDs somewhere.

HHard@Work Ok, that makes sense. I was also just trying to put myself in the shoes of someo...

lambrospetrou•2/10/25, 12:14 PM

that they should be architecting their app for autocleanup/storing IDs somewhere

I still don't agree with this. Autocleaning are very different things from storing IDs.

You almost always want to have a way to get the DO ID from within your application. You cannot always trust users telling you which DO ID they have access to, you need to serve List APIs, and hundreds other use cases. Even if don't store literally the DO IDs, you should have a deterministic way of generating that ID. Otherwise it's a black hole. Like any other storage product, when an entity/account/resource will be deleted you need to do cleanup. Same applies for R2, for DOs, for SQL databases when the IDs are random, and the list goes on.

Autocleaning, not sure where this fits. Maybe just for temporary ephemeral data you want to delete after a TTL, but again, there shouldn't be magic, you design around this deletion.

Maybe it's just me, but I don't want providers to be deleting my data on their own or through something I didn't explicitly configured/coded.

Llambrospetrou > that they should be architecting their app for autocleanup/storing IDs somewhe...

Hard@Work•2/10/25, 12:19 PM

By Autocleaning, I mean like scheduling an alarm that deletes everything stored, if it hasn't been renewed before then.

As an example, in a few of my apps, I use a DO as a more secure cookie. I.e., while the client just sees an opaque value stored, I can ask the DO corresponding to that opaque value, and it returns a user id, their expected user agent/location, etc.. However, because cookies themselves don't persist forever, I want to ensure that the DOs don't stick around forever either, as that would cost me money for no benefit. Thus, I set an alarm for the time that the cookie would expire, and then on every call to the DO, I reset the alarm. If the alarm fires, it deletes all of its storage, so that I'm no longer billed for it

Hard@Work•2/10/25, 12:20 PM

I don't know if this is the way

you are supposed to go about this kind of work, but it seems to work ok for now

Hard@Work•2/10/25, 12:21 PM

For the above case, while I could store every ID I've ever used in a second database somewhere, if I do that, I might as well just not use a DO, and store the metadata in that database.

HHard@Work By Autocleaning, I mean like scheduling an alarm that deletes everything stored,...

sathoro•2/10/25, 12:24 PM

probably cheaper and more efficient to just auto delete after some long period like a few months rather than writing to the alarm every single request

sathoro•2/10/25, 12:25 PM

its not like you are storing a lot of data

Llambrospetrou If you want folks to be able to add/remove DOs on their own without impacting th...

kchro3•2/10/25, 12:25 PM

by "their own worker projects" do you mean making envs per PR / per person? that seems like it would blow up your workers dashboard.

i'm also curious if there are best practices here or if this is what cloudflare does internally?

Ssathoro probably cheaper and more efficient to just auto delete after some long period l...

lambrospetrou•2/10/25, 12:28 PM

Yeah, in this case you are literally talking about ephemeral data. So, setting a TTL for the alarm long enough ahead in time to do the cleanup is the right thing to do. How short or how long you do that it depends on the strictness of your application.

Kkchro3 by "their own worker projects" do you mean making envs per PR / per person? that...

lambrospetrou•2/10/25, 12:29 PM

Well if it's tens of extra projects, it's not a problem. If it's going to be hundreds of thousands then yeah it won't be nice experience. We use separate projects for each of our envs in some teams. Keep it simple.

Llambrospetrou Well if it's tens of extra projects, it's not a problem. If it's going to be hun...

kchro3•2/10/25, 12:33 PM

Do you run into the issue I was describing then where one person would create a DO and then everyone has to pull in those changes in order to deploy?

Kkchro3 Do you run into the issue I was describing then where one person would create a ...

lambrospetrou•2/10/25, 12:42 PM

No, although I am not sure I understood your full issue. In our case we don't deploy per PR. We only deploy after PRs are merged to a staging env, and later in production. So we don't have multiple PRs modifying the same workers.

Llambrospetrou No, although I am not sure I understood your full issue. In our case we don't de...

kchro3•2/10/25, 12:45 PM

ack, I think I might end up doing that too then. We do have a dev vs prod branch, but all of the PRs are deployed to a shared env

Kkchro3 ack, I think I might end up doing that too then. We do have a dev vs prod branch...

lambrospetrou•2/10/25, 12:46 PM

Also, if you are worried about too many script projects, you can extend your CI to delete the deployed workers/DOs once a PR is merged and closed. So you will only have as many projects as your open PRs.
Just something to think about too, if you like having the whole infra per PR.

Llambrospetrou Also, if you are worried about too many script projects, you can extend your CI ...

kchro3•2/10/25, 12:50 PM

ok, thanks. While I have you, I had a couple more questions:

1. for migrations, are you supposed to just have a running changelog of all tags? Or do you delete them after there are no more references to deleted/renamed classes?
2. Do wrangler.toml files support dynamic envs? Like if I wanted a user specific env or a PR specific env, can I have like [[env.$USER.durable_objects]] or something like that?

Kkchro3 ok, thanks. While I have you, I had a couple more questions: 1. for migrations,...

lambrospetrou•2/10/25, 12:51 PM

1. As far as I know yes. You should keep migrations there. I think of them as immutable once deployed.
2. Yes, absolutely. Example: https://github.com/lambrospetrou/rediflare/blob/b2784e0674474d4d1372d89a74be1d3ed9cc63ad/wrangler.toml#L55

Llambrospetrou 1. As far as I know yes. You should keep migrations there. I think of them as im...

kchro3•2/10/25, 12:52 PM

great, thanks so much

Llambrospetrou 1. As far as I know yes. You should keep migrations there. I think of them as im...

kchro3•2/10/25, 12:53 PM

Ah wait, but in the example you shared, the env is hardcoded as staging right?

Kkchro3 Ah wait, but in the example you shared, the env is hardcoded as staging right?

lambrospetrou•2/10/25, 12:55 PM

Yes, but you can generate your own wrangler configuration right before deploying, depending on your CI.
For example, you could have a

wrangler.toml/json

wrangler.toml/json

that has

[env.__REPLACEME__.durable_objects]

[env.__REPLACEME__.durable_objects]

, and you replace that placeholder in your CI with the PR identifier (ie..

PR_ID

PR_ID

), and then run

wrangler deploy -c path/to/wrangler.toml --env=<PR_ID>

wrangler deploy -c path/to/wrangler.toml --env=<PR_ID>

Llambrospetrou Yes, but you can generate your own wrangler configuration right before deploying...

kchro3•2/10/25, 12:57 PM

I see, thanks

Burrito•2/10/25, 8:37 PM

Related, since R2

put

put

accepts a

ReadableStream

ReadableStream

, I was also wonder that if it's possible to implement streaming exporting a DO's storage this way.

Nnil I'm returning a readable stream inside of a response from a durable object. If I...

jed•2/10/25, 8:57 PM

can you use https://developer.mozilla.org/en-US/docs/Web/API/TextEncoderStream ?

MDN Web Docs

TextEncoderStream - Web APIs | MDN

The TextEncoderStream interface of the Encoding API converts a stream of strings into bytes in the UTF-8 encoding. It is the streaming equivalent of TextEncoder.

Llambrospetrou Speculating here since I didn't run your code, but the `env` you get in the DO c...

DiamondDragon•2/10/25, 9:01 PM

Thanks - what I needed to do was pass Env[Bindings]Env[Bindings] to the DO.

else•2/10/25, 9:19 PM

are there plans to expand DO availability to other regions?

else•2/10/25, 9:19 PM

for example, currently there are no DOs spawned near india

Eelse for example, currently there are no DOs spawned near india

1984 Ford Laser•2/10/25, 10:23 PM

India and CF don't exactly get along well

Ben•2/10/25, 10:41 PM

Question, is there an easy way to arbitrary query a durable object sqlite database but restrict it to reads?

For example I need to query raw sql to debug issues in prod, but I don't want it to be easy to be able to accidentally drop rows and tables etc

1984 Ford Laser•2/11/25, 7:28 AM

Does JSON querying work in DO SQLite like it does in D1? What's the main differences between the two implementations that I need to be aware of

11984 Ford Laser Does JSON querying work in DO SQLite like it does in D1? What's the main differe...

lambrospetrou•2/11/25, 10:29 AM

Same.

1984 Ford Laser•2/11/25, 10:32 AM

Do SQL triggers work too?

Avi•2/11/25, 8:21 PM

is there a way to configure my DO/worker to NOT spawn in particular regions?

Avi•2/11/25, 8:22 PM

i'm running into an issue where users in Hong Kong are getting their DO created in that region, but that means OpenAI API calls

I’m wondering though, for cases where you store a little bit of data but then forget to clean it up,

Similar Threads