@lambrospetrou Apologize for pinging, regarding the issue with SQLite storage operations being overc

@lambrospetrou Apologize for pinging, regarding the issue with SQLite storage operations being overcounted, do you have any idea what could be the cause? How should I get this issue to be looked at by Cloudflare? Developers getting charged with these inaccurate counts (over 10x more) is not exactly a light issue.

You can reproduce it by using

npm create cloudflare@latest

npm create cloudflare@latest

hello world template with DO, change to SQLite backend, with the following code: https://gist.github.com/NonSpicyBurrito/f4938faa7a779574d9af945d012f6703. Deploy, send two requests far enough apart so they can show up separately on dashboard, and observe DO metrics:
- First request where migration occurs: 5 rows read, 10 rows written.
- Second request: 26 rows read, 0 row written.
The second request should never result in 26 rows read, there are only 2 rows in the entire database. And the more irrelevant tables you add, the more row reads it counts.

1984 Ford Laser•2/6/25, 7:00 AM

Do SQL triggers work in DOs? I swear someone posted that they'd gotten it working recently

Llambrospetrou If you use DOs, you should not need often to iterate all of them. It defeats the...

Hello, I’m Allie!•2/6/25, 7:11 AM

Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue a delete once used, it won’t propagate instantly, so it might still be usable for up to a minute

HHello, I’m Allie!Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue ...

sathoro•2/6/25, 7:40 AM

yeah better to use a DO w/ an alarm that deletes itself and also destroys the storage upon validation (or sets a flag, either way it gets deleted w/ the alarm)

HHello, I’m Allie!Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue ...

lambrospetrou•2/6/25, 9:11 AM

I don't now what OTP codes meant by the user in that context, since time-based otps don't need to be stored even if I am not mistaken (although been a while since I checked). For the one-time usage, yeah, probably you need some more tracking to enforce that, which you could do with another KV item if needed, or then you go into the DO+alarms territory, but with better modelling.
My point still stands though. If you use DOs, you should design your model such that you don't need to iterate all of them. Otherwise it's not a good fit. If we are talking about a few tens, that's fine, but otherwise it's going to be bad.

BBurrito @lambrospetrou Apologize for pinging, regarding the issue with SQLite storage op...

lambrospetrou•2/6/25, 9:31 AM

Looking at the code, I ran it and using the cursor

rowsRead/rowsWritten

rowsRead/rowsWritten

it's what I would expect. I will share it with the DO team to see if they find anything.

BurritoOP•2/6/25, 9:33 AM

Yeah the cursor numbers seem to be accurate, but dashboard numbers don't match and are way higher.

Llambrospetrou Looking at the code, I ran it and using the cursor `rowsRead/rowsWritten` it's w...

James•2/6/25, 4:12 PM

more importantly, which of these metrics are we being billed for, the ones feeding the dashboard incorrectly, or the cursor?

BurritoOP•2/6/25, 4:35 PM

The cursor numbers are what get billed, and dashboard numbers are supposed to match. It's unclear if the wrong numbers on dashboard are only for display, or if they are actually billed.

JJames more importantly, which of these metrics are we being billed for, the ones feedi...

lambrospetrou•2/6/25, 5:13 PM

BTW, just to make sure people are on the same page. Right now SQLite DOs, do NOT bill you for storage. It's part of the beta program. The team is aware of the discrepancy and will be fixed before actually charging folks.
https://developers.cloudflare.com/durable-objects/platform/pricing/#sqlite-storage-backend

During the initial beta, Storage API billing is not enabled for Durable Object classes using SQLite storage backend. SQLite-backed Durable Objects will incur charges for requests and duration. We plan to enable Storage API billing for Durable Objects using SQLite storage backend in the first half of 2025 after advance notice with the below pricing.

Cloudflare Docs

Pricing · Cloudflare Durable Objects docs

Durable Objects are only available on the Workers Paid plan, and are billed while the Durable Object is active (including the event context).

sup filistine•2/6/25, 7:00 PM

I am running cloudflare/puppeteeer as a client using a browser as a service outside of cloudflare (called Zenrows). The connection over

wss://

wss://

used to work just fine. We only used 1 Durable Object Rpc to do a 50s form filing process using the remote browser. This worked great.

Except as of last week, we are noticing this error like 80% of the time, when we do

puppeteer.connect

puppeteer.connect

Uncaught TypeError: Failed to establish the WebSocket connection: expected server to reply with HTTP status code 101 (switching protocols), but received 520 instead."

Uncaught TypeError: Failed to establish the WebSocket connection: expected server to reply with HTTP status code 101 (switching protocols), but received 520 instead."

Is anyone on the CF staff able to look into this on your end? I can provide my account ID and worker name.

(In talking with Zenrows, they haven't changed anything.)

sup filistine•2/6/25, 7:08 PM

X-posted here as well: workers-and-pages-helpSomething broken with establishing websocket connection with Browser as a service provider

If its better to respond to a post.

HHello, I’m Allie!Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue ...

iLearn•2/6/25, 9:29 PM

Correct and that is the reason i have designed DO per user (identified by mobile number).
BOTH - "one time" and "time based expiration, if unsed" are important in OTP (One Time Passcode" based authentication.

Ssathoro yeah better to use a DO w/ an alarm that deletes itself and also destroys the st...

iLearn•2/6/25, 9:35 PM

valid point.
but, wanted to save this step , as setting alarm + delete .. will eat 2 additional write operations and hence will add up bill (imagine, million OTP based auth happening per day and corresponding cost) .. which, other wise using Wokers-KV platform managed self-destruction will save these 2 steps and cost, as well ..

anyways .. that is more of business perspective .. and not tech side

Llambrospetrou I don't now what OTP codes meant by the user in that context, since time-based o...

iLearn•2/6/25, 9:43 PM

All points valid.
My Qs come from 2 dimensions .. Business + Tech ..
From Tech, ask is design need to be "race-free" + "as much as possible, platform should do house cleaning" .. DOs qualify "race-free" + Workers-KV (qualify house-cleaning)
From Business, ask is "do not burn hole for tasks which are not revenue generators"
(OTP based authentication is not revenue generator)

Anyways, thanks for kind response and sharing your understanding. I am working on design where DOs will be gate-keepers for access to ALL data stored in Workers-KV, and enjoy power of both world.
Shall post more Qs, as i progress and require clarifications.

IiLearn Just to be clear : fact#1 : "A request unit is defined as 4 KB of data read" (as...

iLearn•2/6/25, 9:47 PM

@lambrospetrou Can you please confirm answer for this Q. Thanks,

IiLearn All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech...

lambrospetrou•2/6/25, 11:44 PM

I have a feeling you are overcomplicating things just to avoid doing the cleanup yourself with an alarm and 4 lines of code. You do realize DOs are cheaper 5x than Workers KV for writes, and only 2.5x more expensive for reads, so if you put DOs in the read path just to get the automatic delete from Workers KV, you get the worst of both worlds. And SQLite storage ops are even cheaper.

Anyway, your decision.

https://developers.cloudflare.com/durable-objects/platform/pricing/#storage-api-billing
https://developers.cloudflare.com/workers/platform/pricing/#workers-kv

IiLearn @lambrospetrou Can you please confirm answer for this Q. Thanks,

lambrospetrou•2/6/25, 11:49 PM

According to the docs:

Requests that hit the Durable Objects in-memory cache or that use the multi-key versions of get()/put()/delete() methods are billed the same as if they were a normal, individual request for each key.

https://developers.cloudflare.com/durable-objects/platform/pricing/#key-value-storage-backend

Llambrospetrou I have a feeling you are overcomplicating things just to avoid doing the cleanup...

iLearn•2/7/25, 12:39 AM

[ Following explanation are "general" discussion on DO and NOT in context of my OTP design, (though, principles will be evaluated/applied for OTP design as well, especially the race-condition offering) ]

I, respectfully, differ with statement "get the worst of both worlds".
Reason : "race-condition" management, offering by DO, is "price-less" (imho) . If i put DO as gate-keeper (in front of everything Workers-KV, R2), i make developer life much happier, much code management simpler (plz don't measure with the number of lines) with atleast blocking bugs popping up due to race-conditions while scaling the operations. Which otherwise, there will always a bulk of code to just manage race-conditions.

Also differ with statement "DOs are cheaper".
Reason : Using DOs, effectively, means you are "invoking" workers TWICE. One Master Worker which receives the request and the the DO stub call (which is effectively a worker call).

Also, DO-KV is charged per 4KB read, while Workers-KV is NOT charged based on size of data read/write.
The 5x and 2.5x cheap comparsion flies away with 5x and 2.5x size of data. Means, 20KB (in single write) data is break-even point and anything above this will make DO-KV writes "costlier". Similarly, for Read case.

Llambrospetrou According to the docs: > Requests that hit the Durable Objects in-memory cache o...

iLearn•2/7/25, 12:50 AM

Thanks for confirmation.

IiLearn [ Following explanation are "general" discussion on DO and NOT in context of my...

sathoro•2/7/25, 12:50 AM

you are just storing OTP, why would it be over 4kb of data?

Ssathoro you are just storing OTP, why would it be over 4kb of data?

iLearn•2/7/25, 12:52 AM

please refer last line in P.S.

sathoro•2/7/25, 12:52 AM

you cannot bring up a specific use case and then say "oh actually that doesn't apply because I'm talking about them in general"

options have to be evaluated for specific use cases...

Avi•2/7/25, 12:53 AM

how can I get a request ContextContext inside a DurableObjectDurableObject?

Ssathoro you cannot bring up a specific use case and then say "oh actually that doesn't a...

iLearn•2/7/25, 12:54 AM

agree. my mistake. i did-not put-up my complete use-case and just shared a snippet of the use-case for clarification on DO.

FFlikTeoh Actually KV in DO with SQLite is charged based on sql read pricing, if I underst...

1984 Ford Laser•2/7/25, 4:03 AM

huh I just checked the docs and you're correct. This definitely needs extra clarity

IiLearn All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech...

森

森@VerseLink (RW)•2/7/25, 5:19 AM

I beg to differ, OTP based authentication allows more flexibility to users using your service and thus remove friction or add value to your service as a whole, it doesn't generate revenue directly, but adds then in a indirect manner, the question is not "if this generates revenue" but "is the cost to add this feature important enough that justifies it's cost to implement".

I would also argue that the dev time taken to develop or maintain this is probably more expensive then KV or DO charges. Do what's easier to maintain and develop for your team.

AAvi how can I get a request [`Context`](https://developers.cloudflare.com/workers/ru...

森

森@VerseLink (RW)•2/7/25, 5:20 AM

this.ctxthis.ctx it's part of the DurableObject class

FFlikTeoh Actually KV in DO with SQLite is charged based on sql read pricing, if I underst...

iLearn•2/7/25, 11:10 AM

In the approach of taking "KV in DO with SQLite", limitation of storage of 1GB (as of today) and 10GB (in future, when SQLite becomes GA) will be biggest THREAT-to-SCALE.

While, default KV with DO, has "theoretically" NO LIMIT on storage.

There are multiple dimensions [ COST-to-develop, COST-to-run, COST-to-manage, SCALE-for-PLANET-EARTH (coincidentally, Cloudlare also shares same vision of Planet Earth) ], which a tech-architect-cum-business-leader, need to constantly think.

Basis my preliminary study and information shared by generous insights from tech-leaders in this community, DO has following positioning (my take) :
COST-to-develop : 10/10 (10 ==> Happy Developer and Business)
COST-to-run : 7/10 (Reason to reduce 3 points explained somewhere in above thread, by me) (10==> Happy Business)
COST-to-manage : 10/10 (10 ==> Happy Developer and Business)
SCALE-for-PLANET-EARTH :
02/10 (with SQLite storage "HARD-LIMIT of 10GB in GA")
10/10 (with default KV storage) (10==> Happy Business)

I am extremely bullish for DO capabilities for its "inherent infra" for mananging race condition (though, not verified yet).

My 2 cents. Will update, as i get more insights.

FFlikTeoh > In the approach of taking "KV in DO with SQLite", limitation of storage of 1GB...

iLearn•2/7/25, 11:22 AM

Thanks. Surely will take a look. Keep sharing informative reads.

FFlikTeoh Now we just gotta make sure there's a redundant backup 😅

lambrospetrou•2/7/25, 12:55 PM

You mean backup as in another DO that receives replication updates, or somewhere offline like R2 as actual backup?

BurritoOP•2/7/25, 2:23 PM

Speaking of backup, is there a reference implementation of dumping a SQLite DO's database in a streaming fashion? The implementations I've come across (albeit I wasn't really looking for it) all keep the dump in memory, which wouldn't work for large databases.

m.muiznieks•2/7/25, 3:30 PM

Hello.

From time to time in the logs I see the error "Error: Durable Object exceeded its CPU time limit and was reset."

I am not using the Durable objects directly but I understand that they are behind most of the services here.

I've noticed this at the Zaraz service "context enrich workers" and this happened at the workflows.

None of the processes are related to "resource-heavy" processes.

The question is - when this error hits - does it mean the related data requests are considered lost or they are "reset" and it's safe to think that they will be processed?

BBurrito Speaking of backup, is there a reference implementation of dumping a SQLite DO's...

Brayden•2/7/25, 3:30 PM

If you're looking for a reference as a DIY maybe this PR could help give some ideas (https://github.com/outerbase/starbasedb/pull/76). I still need to run tests on this myself but worth sharing if you're looking at how a solution might work.

BurritoOP•2/7/25, 3:32 PM

Ah nice, didn't know there's a PR for it already, will take a look thanks.

iLearn•2/7/25, 6:06 PM

Any thing similar to D1 "Export an existing D1 database" (https://developers.cloudflare.com/d1/best-practices/import-export-data/#export-an-existing-d1-database) for DO (ONLY Data, NOT Class/Function) with SQLite storage, where .sql file can be dumped per DO to R2/S3 (whenever required) ?

Cloudflare Docs

Import and export data · Cloudflare D1 docs

D1 allows you to import existing SQLite tables and their data directly, enabling you to migrate existing data into D1 quickly and easily. This can be useful when migrating applications to use Workers and D1, or when you want to prototype a schema locally before importing it to your D1 database(s).

Fernando Barroso•2/7/25, 7:33 PM

hey all! I am having issues with durable objects tests using vitest. They work fine locally, but once we run them using GitHub actions we are running into the following problem:

packages/platform-core test: Failed to pop isolated storage stack frame in tests/do/do.test.ts's test "should append events to new stream".
packages/platform-core test: This usually means your Worker tried to access storage outside of a test, or failed to cleanup storage passed across an RPC boundary.
packages/platform-core test: In particular, we were unable to pop Durable Objects storage.
packages/platform-core test: Ensure you `await` all `Promise`s that read or write to these services, and make sure you use the `using` keyword when passing data across JSRPC. See https://developers.cloudflare.com/workers/runtime-apis/rpc/lifecycle#explicit-resource-management for more details.

packages/platform-core test: Failed to pop isolated storage stack frame in tests/do/do.test.ts's test "should append events to new stream".
packages/platform-core test: This usually means your Worker tried to access storage outside of a test, or failed to cleanup storage passed across an RPC boundary.
packages/platform-core test: In particular, we were unable to pop Durable Objects storage.
packages/platform-core test: Ensure you `await` all `Promise`s that read or write to these services, and make sure you use the `using` keyword when passing data across JSRPC. See https://developers.cloudflare.com/workers/runtime-apis/rpc/lifecycle#explicit-resource-management for more details.

has anyone run into this before? I have literally tried every single combination for concurrency and isolation model and I am still running into this.

ᴀ

ᴀkqj•2/7/25, 10:19 PM

Hey team

, I am having issues with DO Error: Durable Object is overloaded. Requests queued for too long.Error: Durable Object is overloaded. Requests queued for too long. .The document says An individual Object has a soft limit of 1,000 requests per second.An individual Object has a soft limit of 1,000 requests per second. is there a setting or configuration which can be set to increase this limit?

ᴀᴀkqj Hey team 👋 , I am having issues with DO `Error: Durable Object is overloaded. ...

Isaac McFadyen•2/7/25, 10:24 PM

No, it's a limitation of the Durable Object infrastructure and is only ever increased on internal Cloudflare applications if extremely needed. The recommended fix is to just use more than 1 Durable Object (which is the use-case DOs are designed for really)

Isaac McFadyen•2/7/25, 10:25 PM

The reason it's "soft" is that it'll heavily depend on how much computation you're doing in a single DO.

Avi•2/7/25, 11:42 PM

is there really no way to catch errors like this in cloudflare?

setTimeout(() => throw new Error("test"), 1000);

setTimeout(() => throw new Error("test"), 1000);

AAvi is there really no way to catch errors like this in cloudflare? ```js setTimeou...

Isaac McFadyen•2/7/25, 11:53 PM

Depends what you want to do with the catch. If you put a try-catch within the timeout itself it'll catch it but because it's in a timeout you can't reply to the client at that point.

IIsaac McFadyen Depends what you want to *do* with the catch. If you put a try-catch within the ...

Avi•2/8/25, 2:05 AM

I mean catching via some sort of global handler, like node has uncaughtExceptionuncaughtException

@lambrospetrou Apologize for pinging, regarding the issue with SQLite storage operations being overc

Similar Threads

Similar Threads

Similar Threads