Our own internal networking.
Our own internal networking.
1Is that the case for all requests from Cloudflare assets/resources to other Cloudflare assets/resources? Like if a DO requests a file from R2 is that also internal networking?
LIn theory it should, yes. I cannot answer for "all" Cloudflare products with certainty though! If it's an internal dev platform product though, it usually stays within our network as much as possible.
Tthanks
1That's cool thank you, always like getting more info about the inner workings. Helps me to be able to appropriately build stuff to take advantage of the platform where possible
LAs said, the DO will move towards the original location/colo if that is healthy for it to run. See durable-objects
LNetwork routing has MANY complexities and edge cases, colos going down, peers going down, cables being bitten by sharks or cut by people, so my suggestion is to benchmark first and not base your assumptions on theoritical statements. 
LStay tuned. The next variant will have ProxyFetchDO spin up Workers to do the fetches. That might be next week though, because I have to do a deep dive on the differences between inheriting from RpcTarget and DurableObject. That may result in optimization of proxy-fetch.
When I'm done with all the variants, I'll upgrade my measurment framework (current 3x more effort than the implementations themselves) and do a big writeup of what I learned.
That said, the theoretical cost savings of this variant is 6x, so not bad for 10s of ms latency add. Think of a chat use case. 95% of the time a post doesn't require an external round trip, but what if your policy was to check links against a pornography or scam listing server before you allow links to go into the thread. Say you have 1000 rooms, you'll have 10s of these per second happen and it won't be all that bursty.
When I'm done with all the variants, I'll upgrade my measurment framework (current 3x more effort than the implementations themselves) and do a big writeup of what I learned.
That said, the theoretical cost savings of this variant is 6x, so not bad for 10s of ms latency add. Think of a chat use case. 95% of the time a post doesn't require an external round trip, but what if your policy was to check links against a pornography or scam listing server before you allow links to go into the thread. Say you have 1000 rooms, you'll have 10s of these per second happen and it won't be all that bursty.
Lport the Schedule API from Agents so we can easily add delays to the proxy-fetch/callback firingCan you elaborate?
LBTW, it handles spikes shockingly well. The DO lets you queue 1000 or so outgoing fetches. Sure, they take a while to all resolve because there are only 6 in flight at a time, but it doesn't crash.
THi, I'm new here and from Malaysia. My timing will be off from you guys but I do have some questions in mind that may sound dumb (i'm only learning BE this year), but I've been trying to understand and use Agent SDK.
I'm trying to build a websocket connection using the Agent SDK framework. I was concern on how is the lifecycle of the agent is going to be? because when reading through the documentation there are 2 types of Websockets connection and was suggesting to use the Hibernation Websocket.
With the Agent SDK basically handling everything even on the frontend with useAgent() and backend handling all the routing to the unique agent I will be creating, does the durable object goes into hibernation / sleep when the the agent websocket is onClose() ? Or it will be a default durable object behavior where it still wait for the 10s before hibernation?
I'm trying to build a websocket connection using the Agent SDK framework. I was concern on how is the lifecycle of the agent is going to be? because when reading through the documentation there are 2 types of Websockets connection and was suggesting to use the Hibernation Websocket.
With the Agent SDK basically handling everything even on the frontend with useAgent() and backend handling all the routing to the unique agent I will be creating, does the durable object goes into hibernation / sleep when the the agent websocket is onClose() ? Or it will be a default durable object behavior where it still wait for the 10s before hibernation?
LWhile it is an option in PartyKit, upon which Agents is built, Agents always sets up a hibernating WebSocket.
PJust updated
wranglerHow can I DELETE a durable object instance?
LWhat do you expect to see? You cannot interact with individual Durable Objects outside of a worker (in addition to the just launched Studio in the dashboard).
LYou don't delete DOs. You can clear their storage by calling the
storage.deleteAll()deleteAlarm()Hmm, trying to see if I can use Cloudflare containers for single use jobs, but their are tied to DOs, so I want to delete the container instance after use. Maybe I need to use the storage.deleteAll() and deleteAlam() to clear the DO context then I can reuse that slot?
PI was trying to figure out how to delete a durable object instance, because I have container images that are just piling up taking tons of space (probably), I actaully don't know, cant see that information.
LAsking in #containers-beta will give you better answers on how to clean up resources created for containers.
Pactually no one from Cloudflare answers anything there, I've been trying for a few days now. This is how I've started venturing into these other places
LI doubt the DOs hold any image space. Check out the wrangler commands if you want to delete images: https://developers.cloudflare.com/workers/wrangler/commands/#containers
PSorry I meant to say "instance", not "image"... every container execution loads an image into an instance. When the instance is finished, it just sits around. That is the thing I'm trying to clean up and I think it's the thing associated with a DO.
LI still don't understand why you want to clean it up? If you don't use it, it will disappear, isn't it? I haven't followed much on containers but it seems weird to me that you want to delete a DO to cleanup some container image space.
PIn this particular use case, it's kind of a security best practice, these are ephermal github runners, so they run once and they should be completeley deleted after
LThe DO instance itself doesn't have the image in its storage, the image is loaded elsewhere and the DO is just a coordinator for the container.
Pya that's what I was afraid of, I have read the actual region (datacenter) a container instance runs is not necessarily the same as the DO, which leads me to believe the DO storage clearing, might NOT actually delete the container instance
1Are you not able to spin up more instances? When a container is shown as inactive in the dash I dont think it actually exists anymore, and should disappear on its own eventually
PI'm seeing them just stick around, for 4 days now, have't tested any longer.
1Yeah they dont still exist, they arent taking up any space. I think youre stressing over nothing
PWell if you are building a real SaaS, and making claims about the way ephemeral compute environments actually dispose of data after use, you want to be 100% sure
PThe State = Inactive actually leads me to belive these instance could actually be re-hydrated, so their state must still exist
1Because you can reboot them from scratch
1Here's some docs on this topic: https://developers.cloudflare.com/containers/platform-details/architecture/#container-shutdown
Persistent disk
All disk is ephemeral. When a Container instance goes to sleep, the next time it is started, it will have a fresh disk as defined by its container image. Persistent disk is something the Cloudflare team is exploring in the future, but is not slated for the near term.Ok, so it's not like "paused" where the container instance is somehow frozen
1No it delete
POk thanks @1984 Ford Laser for the clarification. I just wanted to make sure I wasn't going to rack petabytes of frozen container instances just waiting to be re-hydrated. But sounds like that is not how it works, anything in a State of Inactive is actually just totally deleted.
1You can track your active usage in the Metrics tab of the container dash
PYes the usage shows what I expect, just paying for the execution time. I just want to be 100% sure I know how the data is handled for these ephemeral github runner use cases, so I can explain it to others. I can imagine this could be a security question in a architecture review/deep dive
PHey @Gabi | Containers and team - follow-up from my ephemeral GitHub runners discussion.
Confirmed Understanding:
Container disk is ephemeral and deleted on stop (thanks for clarifying!)
Only paying for active execution time
Inactive containers don't consume storage
Remaining Question:
While the container instances clean up properly, the Durable Object instances themselves accumulate in the namespace indefinitely.
Current Pattern:
Confirmed Understanding:
Container disk is ephemeral and deleted on stop (thanks for clarifying!) Only paying for active execution time Inactive containers don't consume storageRemaining Question:
While the container instances clean up properly, the Durable Object instances themselves accumulate in the namespace indefinitely.
Current Pattern:
- Each job:
→ unique DO createdidFromName('job-{jobId}') - Container runs → stops → DO hibernates (but persists)
- At 1000 jobs/day → 30K DOs/month → 365K DOs/year**Dashboard shows all these inactive DOs forever** (see screenshot from last week).**Questions:**
- Is this accumulation pattern expected/acceptable for ephemeral workloads?
- Do hibernated DOs (with empty storage via deleteAll()) have any cost or limits?
- Should I implement DO pooling to reuse ~100 DOs across all jobs?
- Or just call storage.deleteAll() + deleteAlarm() in onStop() and not worry about it?For context: GitHub Actions runners are true ephemeral workloads - each job should leave zero trace. Want to ensure I'm architecting this correctly for production SaaS.Current scale: 10 concurrent, but could scale up to 100-500 as usage grows.Thanks!
LDo hibernated DOs (with empty storage via deleteAll()) have any cost or limits?No.
You cannot remove DOs from the namespace list at the moment. It's not a containers thing, it's a DOs thing.
1Are you storing anything in the DO storage currently? If you dont utilise storage in any way currently then they wont incur any storage costs at all
PI don't think so but all of this is kind of magical so I don't actually always know what's happening behind the scenes, like are the container logs being stored as part of the DO storage? I don't know.
I'm coming form a cloud background, where I've seen lots of users get burnned becuase of massive CloudWatch logs bills for example.
I'm coming form a cloud background, where I've seen lots of users get burnned becuase of massive CloudWatch logs bills for example.
1Cloudflare products and the runtime definitely feel like magic sometimes. Logs don't cost anything to just have them emitted, there are products like Workers Analytics Engine or Logpush that do cost money, but simply having them be logged and viewing them live is free
PWill I run into scalability issues, like just paging or searching my DO's if I have potentially millions of them? If they are basically ALL not used, it just seems wasteful. This is why I was thinking I might want to implement so kind of Pooling scheme, where I have a fixed number of DOs that get reused to spin up new container instances, this way I only ever have like 100 DOs.
1Yeah I wouldn't ever worry about having too many instances, especially if you aren't using any storage in them. There isn't any account limits to the number of DO instances you spin up, and theoretically you can scale up millions instantly. There's some mental blocks to jump over with DOs, I would focus on building before worrying about runtime stuff that basically isn't an issue
LI think there is a small mental model mismatch going on here. I never get the full list of DOs and walk it for just the reason you mention. Rather, the way I think everyone does it, is we keep a list of active ones somewhere else. This can either be a stringified version of the uniqueId, or more commonly, folks use getByName or the two-step equivalent of that (get the id by name, then get the stub from that id). The "name" is something we already use for other purposes in most cases. For instance, I use Stytch which creates a unique id for each organization in our SaaS product. That's the "name" I use for my OrgDO. How does that sit with you?
PThanks Larry, this is inline with my understanding of best practices, thinking critically about the "partitioning" strategy for DO, basically how you uniquely name DOs. For named/identified things like organizations, users, cities, etc. this is perfect. The thing I'm struggling with is best practice for using DO's with single use identifiers (like a job run id). I think I just need to accept there will be lots of DO's lingering around forever or use a pool of 100x DOs like "runner-001", "runner-002", etc. and then just reuse those, so my OCD isn't triggered 
- My spider sense is telling me be careful about letting millions of resources just pile up, it could come back to bite you - but I get lots of feedback on this forum saying "don't worry about it"
POr maybe I need to think more about the partition key that actually might make sense for this use case, the account and repo and workflow name for example could be concatenated to create a unique slot. I could either use that or a hash of that to create a unique identifier. - this is probably the way...