All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech, ask is design need

All points valid.
My Qs come from 2 dimensions .. Business + Tech ..
From Tech, ask is design need to be "race-free" + "as much as possible, platform should do house cleaning" .. DOs qualify "race-free" + Workers-KV (qualify house-cleaning)
From Business, ask is "do not burn hole for tasks which are not revenue generators"
(OTP based authentication is not revenue generator)

Anyways, thanks for kind response and sharing your understanding. I am working on design where DOs will be gate-keepers for access to ALL data stored in Workers-KV, and enjoy power of both world.
Shall post more Qs, as i progress and require clarifications.

IiLearn Just to be clear : fact#1 : "A request unit is defined as 4 KB of data read" (as...

iLearnOP•2/6/25, 9:47 PM

@lambrospetrou Can you please confirm answer for this Q. Thanks,

IiLearn All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech...

lambrospetrou•2/6/25, 11:44 PM

I have a feeling you are overcomplicating things just to avoid doing the cleanup yourself with an alarm and 4 lines of code. You do realize DOs are cheaper 5x than Workers KV for writes, and only 2.5x more expensive for reads, so if you put DOs in the read path just to get the automatic delete from Workers KV, you get the worst of both worlds. And SQLite storage ops are even cheaper.

Anyway, your decision.

https://developers.cloudflare.com/durable-objects/platform/pricing/#storage-api-billing
https://developers.cloudflare.com/workers/platform/pricing/#workers-kv

IiLearn @lambrospetrou Can you please confirm answer for this Q. Thanks,

lambrospetrou•2/6/25, 11:49 PM

According to the docs:

Requests that hit the Durable Objects in-memory cache or that use the multi-key versions of get()/put()/delete() methods are billed the same as if they were a normal, individual request for each key.

https://developers.cloudflare.com/durable-objects/platform/pricing/#key-value-storage-backend

Llambrospetrou I have a feeling you are overcomplicating things just to avoid doing the cleanup...

iLearnOP•2/7/25, 12:39 AM

[ Following explanation are "general" discussion on DO and NOT in context of my OTP design, (though, principles will be evaluated/applied for OTP design as well, especially the race-condition offering) ]

I, respectfully, differ with statement "get the worst of both worlds".
Reason : "race-condition" management, offering by DO, is "price-less" (imho) . If i put DO as gate-keeper (in front of everything Workers-KV, R2), i make developer life much happier, much code management simpler (plz don't measure with the number of lines) with atleast blocking bugs popping up due to race-conditions while scaling the operations. Which otherwise, there will always a bulk of code to just manage race-conditions.

Also differ with statement "DOs are cheaper".
Reason : Using DOs, effectively, means you are "invoking" workers TWICE. One Master Worker which receives the request and the the DO stub call (which is effectively a worker call).

Also, DO-KV is charged per 4KB read, while Workers-KV is NOT charged based on size of data read/write.
The 5x and 2.5x cheap comparsion flies away with 5x and 2.5x size of data. Means, 20KB (in single write) data is break-even point and anything above this will make DO-KV writes "costlier". Similarly, for Read case.

Llambrospetrou According to the docs: > Requests that hit the Durable Objects in-memory cache o...

iLearnOP•2/7/25, 12:50 AM

Thanks for confirmation.

IiLearn [ Following explanation are "general" discussion on DO and NOT in context of my...

sathoro•2/7/25, 12:50 AM

you are just storing OTP, why would it be over 4kb of data?

Ssathoro you are just storing OTP, why would it be over 4kb of data?

iLearnOP•2/7/25, 12:52 AM

please refer last line in P.S.

sathoro•2/7/25, 12:52 AM

you cannot bring up a specific use case and then say "oh actually that doesn't apply because I'm talking about them in general"

options have to be evaluated for specific use cases...

Avi•2/7/25, 12:53 AM

how can I get a request ContextContext inside a DurableObjectDurableObject?

Ssathoro you cannot bring up a specific use case and then say "oh actually that doesn't a...

iLearnOP•2/7/25, 12:54 AM

agree. my mistake. i did-not put-up my complete use-case and just shared a snippet of the use-case for clarification on DO.

FFlikTeoh Actually KV in DO with SQLite is charged based on sql read pricing, if I underst...

1984 Ford Laser•2/7/25, 4:03 AM

huh I just checked the docs and you're correct. This definitely needs extra clarity

IiLearn All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech...

森

森@VerseLink (RW)•2/7/25, 5:19 AM

I beg to differ, OTP based authentication allows more flexibility to users using your service and thus remove friction or add value to your service as a whole, it doesn't generate revenue directly, but adds then in a indirect manner, the question is not "if this generates revenue" but "is the cost to add this feature important enough that justifies it's cost to implement".

I would also argue that the dev time taken to develop or maintain this is probably more expensive then KV or DO charges. Do what's easier to maintain and develop for your team.

AAvi how can I get a request [`Context`](https://developers.cloudflare.com/workers/ru...

森

森@VerseLink (RW)•2/7/25, 5:20 AM

this.ctxthis.ctx it's part of the DurableObject class

FFlikTeoh Actually KV in DO with SQLite is charged based on sql read pricing, if I underst...

iLearnOP•2/7/25, 11:10 AM

In the approach of taking "KV in DO with SQLite", limitation of storage of 1GB (as of today) and 10GB (in future, when SQLite becomes GA) will be biggest THREAT-to-SCALE.

While, default KV with DO, has "theoretically" NO LIMIT on storage.

There are multiple dimensions [ COST-to-develop, COST-to-run, COST-to-manage, SCALE-for-PLANET-EARTH (coincidentally, Cloudlare also shares same vision of Planet Earth) ], which a tech-architect-cum-business-leader, need to constantly think.

Basis my preliminary study and information shared by generous insights from tech-leaders in this community, DO has following positioning (my take) :
COST-to-develop : 10/10 (10 ==> Happy Developer and Business)
COST-to-run : 7/10 (Reason to reduce 3 points explained somewhere in above thread, by me) (10==> Happy Business)
COST-to-manage : 10/10 (10 ==> Happy Developer and Business)
SCALE-for-PLANET-EARTH :
02/10 (with SQLite storage "HARD-LIMIT of 10GB in GA")
10/10 (with default KV storage) (10==> Happy Business)

I am extremely bullish for DO capabilities for its "inherent infra" for mananging race condition (though, not verified yet).

My 2 cents. Will update, as i get more insights.

FFlikTeoh > In the approach of taking "KV in DO with SQLite", limitation of storage of 1GB...

iLearnOP•2/7/25, 11:22 AM

Thanks. Surely will take a look. Keep sharing informative reads.

FFlikTeoh Now we just gotta make sure there's a redundant backup 😅

lambrospetrou•2/7/25, 12:55 PM

You mean backup as in another DO that receives replication updates, or somewhere offline like R2 as actual backup?

Burrito•2/7/25, 2:23 PM

Speaking of backup, is there a reference implementation of dumping a SQLite DO's database in a streaming fashion? The implementations I've come across (albeit I wasn't really looking for it) all keep the dump in memory, which wouldn't work for large databases.

m.muiznieks•2/7/25, 3:30 PM

Hello.

From time to time in the logs I see the error "Error: Durable Object exceeded its CPU time limit and was reset."

I am not using the Durable objects directly but I understand that they are behind most of the services here.

I've noticed this at the Zaraz service "context enrich workers" and this happened at the workflows.

None of the processes are related to "resource-heavy" processes.

The question is - when this error hits - does it mean the related data requests are considered lost or they are "reset" and it's safe to think that they will be processed?

BBurrito Speaking of backup, is there a reference implementation of dumping a SQLite DO's...

Brayden•2/7/25, 3:30 PM

If you're looking for a reference as a DIY maybe this PR could help give some ideas (https://github.com/outerbase/starbasedb/pull/76). I still need to run tests on this myself but worth sharing if you're looking at how a solution might work.

Burrito•2/7/25, 3:32 PM

Ah nice, didn't know there's a PR for it already, will take a look thanks.

iLearnOP•2/7/25, 6:06 PM

Any thing similar to D1 "Export an existing D1 database" (https://developers.cloudflare.com/d1/best-practices/import-export-data/#export-an-existing-d1-database) for DO (ONLY Data, NOT Class/Function) with SQLite storage, where .sql file can be dumped per DO to R2/S3 (whenever required) ?

Cloudflare Docs

Import and export data · Cloudflare D1 docs

D1 allows you to import existing SQLite tables and their data directly, enabling you to migrate existing data into D1 quickly and easily. This can be useful when migrating applications to use Workers and D1, or when you want to prototype a schema locally before importing it to your D1 database(s).

Fernando Barroso•2/7/25, 7:33 PM

hey all! I am having issues with durable objects tests using vitest. They work fine locally, but once we run them using GitHub actions we are running into the following problem:

packages/platform-core test: Failed to pop isolated storage stack frame in tests/do/do.test.ts's test "should append events to new stream".
packages/platform-core test: This usually means your Worker tried to access storage outside of a test, or failed to cleanup storage passed across an RPC boundary.
packages/platform-core test: In particular, we were unable to pop Durable Objects storage.
packages/platform-core test: Ensure you `await` all `Promise`s that read or write to these services, and make sure you use the `using` keyword when passing data across JSRPC. See https://developers.cloudflare.com/workers/runtime-apis/rpc/lifecycle#explicit-resource-management for more details.

packages/platform-core test: Failed to pop isolated storage stack frame in tests/do/do.test.ts's test "should append events to new stream".
packages/platform-core test: This usually means your Worker tried to access storage outside of a test, or failed to cleanup storage passed across an RPC boundary.
packages/platform-core test: In particular, we were unable to pop Durable Objects storage.
packages/platform-core test: Ensure you `await` all `Promise`s that read or write to these services, and make sure you use the `using` keyword when passing data across JSRPC. See https://developers.cloudflare.com/workers/runtime-apis/rpc/lifecycle#explicit-resource-management for more details.

has anyone run into this before? I have literally tried every single combination for concurrency and isolation model and I am still running into this.

ᴀ

ᴀkqj•2/7/25, 10:19 PM

Hey team

, I am having issues with DO

Error: Durable Object is overloaded. Requests queued for too long.

Error: Durable Object is overloaded. Requests queued for too long.

.The document says

An individual Object has a soft limit of 1,000 requests per second.

An individual Object has a soft limit of 1,000 requests per second.

is there a setting or configuration which can be set to increase this limit?

ᴀᴀkqj Hey team 👋 , I am having issues with DO `Error: Durable Object is overloaded. ...

Isaac McFadyen•2/7/25, 10:24 PM

No, it's a limitation of the Durable Object infrastructure and is only ever increased on internal Cloudflare applications if extremely needed. The recommended fix is to just use more than 1 Durable Object (which is the use-case DOs are designed for really)

Isaac McFadyen•2/7/25, 10:25 PM

The reason it's "soft" is that it'll heavily depend on how much computation you're doing in a single DO.

Avi•2/7/25, 11:42 PM

is there really no way to catch errors like this in cloudflare?

setTimeout(() => throw new Error("test"), 1000);

setTimeout(() => throw new Error("test"), 1000);

AAvi is there really no way to catch errors like this in cloudflare? ```js setTimeou...

Isaac McFadyen•2/7/25, 11:53 PM

Depends what you want to do with the catch. If you put a try-catch within the timeout itself it'll catch it but because it's in a timeout you can't reply to the client at that point.

IIsaac McFadyen Depends what you want to *do* with the catch. If you put a try-catch within the ...

Avi•2/8/25, 2:05 AM

I mean catching via some sort of global handler, like node has uncaughtExceptionuncaughtException

Avi•2/8/25, 2:06 AM

if this happens in the codebase and someone forget to explicitly try-catch it, there is afaik zero way to observe or react to that

Avi•2/8/25, 2:06 AM

especially since it might not be in the context of a request as in a DO

AAvi if this happens in the codebase and someone forget to explicitly try-catch it, t...

Isaac McFadyen•2/8/25, 2:06 AM

You can use a Tail Worker: https://developers.cloudflare.com/workers/observability/logs/tail-workers/

Isaac McFadyen•2/8/25, 2:07 AM

A Tail Worker receives information about the execution of other Workers (known as producer Workers), such as HTTP statuses, data passed to console.log() or uncaught exceptions

Isaac McFadyen•2/8/25, 2:07 AM

I believe you can actually have it be the same worker

Isaac McFadyen•2/8/25, 2:07 AM

Just add a tailtail handler and in the wrangler.json/wrangler.toml, point the Worker to itself.

IIsaac McFadyen You can use a Tail Worker: https://developers.cloudflare.com/workers/observabili...

Avi•2/8/25, 2:08 AM

its a very poor solution for DOs, because tail worker only process on requests, and the HTTP req/response lifecycle often does not map cleanly onto DOs (that's the whole point of DOs!)

Isaac McFadyen•2/8/25, 2:08 AM

Ah I see

Isaac McFadyen•2/8/25, 2:08 AM

Not sure then.

Avi•2/8/25, 2:09 AM

seems like the "right" solutions would be for CF to either:
- implement addEventListener('uncaughtexception', ...)addEventListener('uncaughtexception', ...) like node does
or
- enable tailworkers to operate more granularly on the stream of events from a worker (e.g. console.log events in real time)

Rahul gangotri•2/8/25, 2:57 PM

Actually i was building a websocket product on durable objects for a pub sub based models

With channels like public, private, presence, cache

I create user apps where each user can create multiple apps and each app is a separate DO

I just want to have a quick chat with you to know that can i make this a saas model with just having 1 namespace and sell to users?

Is that technically possible? I know storage with be a limit, but i just need to store channel and its connected clients and not the messages

Rahul gangotri•2/8/25, 6:22 PM

anyone? From cloudflare?

Please reply

IiLearn Any thing similar to D1 "Export an existing D1 database" (https://developers.cl...

iLearnOP•2/8/25, 6:40 PM

bumping on top for response

All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech, ask is design need

Similar Threads

Similar Threads

Similar Threads