Is there a faster way to debug DO storage metrics than waiting tens of minutes for dashboard to upda

Is there a faster way to debug DO storage metrics than waiting tens of minutes for dashboard to update? Is there a way to do it locally?

BBurrito Is there a faster way to debug DO storage metrics than waiting tens of minutes f...

Hard@Work•2/5/25, 6:05 PM

I don't think so? IIRC the dashboard is powered by GraphQL, which you can query directly, however it would still have delays

BurritoOP•2/5/25, 6:06 PM

Well that's problematic.

BBurrito Well that's problematic.

Hard@Work•2/5/25, 6:07 PM

It's a difficult thing because while seeing storage metrics for a single DO might be easy, seeing aggregated metrics for millions/billions of DOs fast is very difficult

BurritoOP•2/5/25, 6:08 PM

Yeah that's understandable, but I would've thought there's a way to do it locally, since the runtime is now open source and all that.

BBurrito The request results in 236 row reads in DO metrics, but worker logs only show 5 ...

BurritoOP•2/5/25, 6:09 PM

I'm trying to debug this issue and it's really not helping that I need to wait a long time before I can observe what had happened.

BBurrito Yeah that's understandable, but I would've thought there's a way to do it locall...

Hard@Work•2/5/25, 6:09 PM

Ooh, ok. I'm not entirely sure, is the DO storage engine actually open source?

Hard@Work•2/5/25, 6:10 PM

I don't remember if it uses a stub within Miniflare

BurritoOP•2/5/25, 6:10 PM

Ah not sure then.

BurritoOP•2/5/25, 6:19 PM

Well I've tried a reproduction, starting from

create cloudflare

create cloudflare

template with modified code, SQLite backend:

import { DurableObject } from 'cloudflare:workers';

export class MyDurableObject extends DurableObject<Env> {
    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env);

        ctx.blockConcurrencyWhile(async () => {
            await ctx.storage.transaction(async () => {
                const version = await ctx.storage.get('migrationVersion');
                if (version === 1) return;

                ctx.storage.sql.exec(`
                    CREATE TABLE Configs (
                        key TEXT NOT NULL,
                        value TEXT NOT NULL,
                        PRIMARY KEY (key)
                    )
                    STRICT;
                `);
                ctx.storage.sql.exec(`INSERT INTO Configs (key, value) VALUES (?, ?);`, 'foo', 'bar');

                await ctx.storage.put('migrationVersion', 1);
            });
        });
    }

    async sayHello(key: string): Promise<string> {
        const { value } = this.ctx.storage.sql.exec(`SELECT value FROM Configs WHERE key = ?;`, key).one();

        return `Hello, ${value}!`;
    }
}

export default {
    async fetch(request, env, ctx): Promise<Response> {
        const url = new URL(request.url);
        if (url.pathname !== '/hello-world') return new Response(null, { status: 404 });

        let id: DurableObjectId = env.MY_DURABLE_OBJECT.idFromName('hello-world');
        let stub = env.MY_DURABLE_OBJECT.get(id);
        let greeting = await stub.sayHello('foo');
        return new Response(greeting);
    },
} satisfies ExportedHandler<Env>;

import { DurableObject } from 'cloudflare:workers';

export class MyDurableObject extends DurableObject<Env> {
    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env);

        ctx.blockConcurrencyWhile(async () => {
            await ctx.storage.transaction(async () => {
                const version = await ctx.storage.get('migrationVersion');
                if (version === 1) return;

                ctx.storage.sql.exec(`
                    CREATE TABLE Configs (
                        key TEXT NOT NULL,
                        value TEXT NOT NULL,
                        PRIMARY KEY (key)
                    )
                    STRICT;
                `);
                ctx.storage.sql.exec(`INSERT INTO Configs (key, value) VALUES (?, ?);`, 'foo', 'bar');

                await ctx.storage.put('migrationVersion', 1);
            });
        });
    }

    async sayHello(key: string): Promise<string> {
        const { value } = this.ctx.storage.sql.exec(`SELECT value FROM Configs WHERE key = ?;`, key).one();

        return `Hello, ${value}!`;
    }
}

export default {
    async fetch(request, env, ctx): Promise<Response> {
        const url = new URL(request.url);
        if (url.pathname !== '/hello-world') return new Response(null, { status: 404 });

        let id: DurableObjectId = env.MY_DURABLE_OBJECT.idFromName('hello-world');
        let stub = env.MY_DURABLE_OBJECT.get(id);
        let greeting = await stub.sayHello('foo');
        return new Response(greeting);
    },
} satisfies ExportedHandler<Env>;

I've sent two requests far enough apart to make sure they show up separately on dashboard, and first request shows 5 rows read + 8 rows written, second request shows 8 rows read + 0 row written.

BurritoOP•2/5/25, 6:21 PM

It's not as outlandish as my actual project, but these numbers still don't make any sense. I fail to see how this code can result in that many reads or writes.

James•2/5/25, 6:32 PM

i bet its something goofy w the psuedo KV api on Sqlite since its abstracting the real SQL away for you

BurritoOP•2/5/25, 6:59 PM

Okay we are getting somewhere, if you replace the migration with:

CREATE TABLE Configs (
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (key)
)
STRICT;

CREATE TABLE Config1 (
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (key)
)
STRICT;

CREATE TABLE Config2 (
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (key)
)
STRICT;

-- Config3, Config4, Config5

CREATE TABLE Configs (
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (key)
)
STRICT;

CREATE TABLE Config1 (
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (key)
)
STRICT;

CREATE TABLE Config2 (
    key TEXT NOT NULL,
    value TEXT NOT NULL,
    PRIMARY KEY (key)
)
STRICT;

-- Config3, Config4, Config5

Where

Config1

Config1

Config5

Config5

are all useless tables, and keeping rest of the code the same.
Now repeat the same 2 requests:
- First request: 8 written, 5 read.
- Second request: 0 written, 28 read.

BurritoOP•2/5/25, 7:00 PM

@lambrospetrou ^ Is the above to be expected?

iLearn•2/5/25, 7:09 PM

Need help (in my design) :

Facts :
F#1 : I shall have 1 Class, say UserOTPStore. This will will store 6-digit OTP as Key and Timestamp (of expiration) as Value.
F#2 : User shall be identified by "mobile number"

Implementation :
1. User requests OTP. Create DO with "mobile_number" and store OTP and Timestamp inside respective DO.

Please confirm my understanding for following Qs :

Q#1 : How to fetch ALL user for whom DOs has been created (with persistent data) ?
A#1 : Use DO-API : "/accounts/{account_id}/workers/durable_objects/namespaces/{id}/objects"

Q#2 : How to fetch data stored for ALL Users :
A#2 : Loop through the list of DO-IDs received from A#1, and call stub for each DO-ID (assume a function dump_stored_data() is part of the UserOTPStore Class)

Q#3 : How to clean-up / delete ALL Users data ? (basically, remove everything related to DO UserOTPStore Class)
A#3 : NO simple way. Need to use approach for A#2.
PLEASE suggest a simple O(1) operation, (as approach taken for delete based on A#2 will be really costly)

Thanks in advance for kind consideration in answering/confirming my Qs.

BBurrito Okay we are getting somewhere, if you replace the migration with: ```sql CREATE ...

James•2/5/25, 7:37 PM

Thats so bizarre, what if you vary this one more time, and take out the migration get/put.

IiLearn Need help (in my design) : Facts : F#1 : I shall have 1 Class, say UserOTPStor...

James•2/5/25, 7:39 PM

you're better off storing a reference to the stubs centrally in another store/D1/DO/whatever rather than using the DO-API.

JJames you're better off storing a reference to the stubs centrally in another store/D1...

iLearn•2/5/25, 7:44 PM

Ok. Thanks for suggestion.
Any reason for not encouraging to use DO-API ? (actually want to offload as much as possible to platform , instead of house-keeping self)

James•2/5/25, 7:48 PM

lots of reasons, the DO API is weak if you want to actually find specific DOs and not iterate over all the things, you're better of keeping central ref plus useful business data, in your case like, i dont know, lastSMSTs or something

I generally keep DO data contained in the DO if it supports the DO user/client directly in thier specific isolated tasks, and anything system related, gets kept a level above the DO for central management, cleanup, aggregation etc.

JJames lots of reasons, the DO API is weak if you want to actually find specific DOs an...

iLearn•2/5/25, 7:52 PM

Ahh. Nice Point and makes sense.
QQ : (my Q#3 above) How to clean-up / delete ALL Users data ? (basically, remove everything related to DO UserOTPStore Class) ?

JJames Thats so bizarre, what if you vary this one more time, and take out the migratio...

BurritoOP•2/5/25, 7:56 PM

I don't think that matters anyways, since the migration code won't run after the first request.

BurritoOP•2/5/25, 7:58 PM

The more bizarre part is that row read somehow scales with the amount of tables, in the first reproduction there were 2 tables (

Configs

Configs

, and the hidden KV API table) and second request costs 8 reads, in the second reproduction there were 7 tables (

Configs

Configs

Config1

Config1

Config5

Config5

, and hidden KV API table) and the second request costs 28 reads.

BurritoOP•2/5/25, 7:58 PM

This seems like a bug to me.

BBurrito I don't think that matters anyways, since the migration code won't run after the...

James•2/5/25, 8:01 PM

id still try it, there is 1 get, and who knows if the get is doing something funny w schema reads

BurritoOP•2/5/25, 8:03 PM

Sure I'll try it after this. It'd be much easier to test these things if it didn't take half an hour to do one test because things show up so slowly on dashboard.

JJames id still try it, there is 1 get, and who knows if the get is doing something fun...

BurritoOP•2/5/25, 8:52 PM

Nope, removing the migration code afterwards drops the rows read from 28 to 27.

BurritoOP•2/5/25, 8:53 PM

There's something weird going on.

BurritoOP•2/5/25, 10:42 PM

I've completely removed KV API from my SQLite DO, and it's still showing 234 row reads in dashboard metrics, despite logging shows the 6 queries only have 1 row read each. There is something very wrong with either the metrics, or how row read/written are counted.

IiLearn Ok. Thanks for suggestion. Any reason for not encouraging to use DO-API ? (actua...

lambrospetrou•2/5/25, 11:04 PM

If you use DOs, you should not need often to iterate all of them. It defeats the purpose of using DOs in the first place.
For the OTP example, if you want to offload OTP expiration to the platform, and deletion, etc, why not just use Workers KV and be done with it?
Set your expiration and let everything be deleted. OTP codes are immutable, so you write a key like

<mobile-phone>-<OTP>

<mobile-phone>-<OTP>

and value being the timestamp or whatever info you want. When the request comes in you can check the specific OTP if it exists or not indicating validity, and search by prefix the phone number.

If that satisfies your needs, don't use DOs, since your access patterns requiring ALL DOs data is very bad for cost and performance.

Also, as others mentioned before, usually it's good to do some bookkeeping of DOs created in some other DOs so that you can do faster search/listing of them. Read https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/ for more details on that.

BurritoOP•2/6/25, 12:44 AM

@lambrospetrou Apologize for pinging, regarding the issue with SQLite storage operations being overcounted, do you have any idea what could be the cause? How should I get this issue to be looked at by Cloudflare? Developers getting charged with these inaccurate counts (over 10x more) is not exactly a light issue.

You can reproduce it by using

npm create cloudflare@latest

npm create cloudflare@latest

hello world template with DO, change to SQLite backend, with the following code: https://gist.github.com/NonSpicyBurrito/f4938faa7a779574d9af945d012f6703. Deploy, send two requests far enough apart so they can show up separately on dashboard, and observe DO metrics:
- First request where migration occurs: 5 rows read, 10 rows written.
- Second request: 26 rows read, 0 row written.
The second request should never result in 26 rows read, there are only 2 rows in the entire database. And the more irrelevant tables you add, the more row reads it counts.

1984 Ford Laser•2/6/25, 7:00 AM

Do SQL triggers work in DOs? I swear someone posted that they'd gotten it working recently

Llambrospetrou If you use DOs, you should not need often to iterate all of them. It defeats the...

Hello, I’m Allie!•2/6/25, 7:11 AM

Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue a delete once used, it won’t propagate instantly, so it might still be usable for up to a minute

HHello, I’m Allie!Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue ...

sathoro•2/6/25, 7:40 AM

yeah better to use a DO w/ an alarm that deletes itself and also destroys the storage upon validation (or sets a flag, either way it gets deleted w/ the alarm)

HHello, I’m Allie!Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue ...

lambrospetrou•2/6/25, 9:11 AM

I don't now what OTP codes meant by the user in that context, since time-based otps don't need to be stored even if I am not mistaken (although been a while since I checked). For the one-time usage, yeah, probably you need some more tracking to enforce that, which you could do with another KV item if needed, or then you go into the DO+alarms territory, but with better modelling.
My point still stands though. If you use DOs, you should design your model such that you don't need to iterate all of them. Otherwise it's not a good fit. If we are talking about a few tens, that's fine, but otherwise it's going to be bad.

BBurrito @lambrospetrou Apologize for pinging, regarding the issue with SQLite storage op...

lambrospetrou•2/6/25, 9:31 AM

Looking at the code, I ran it and using the cursor

rowsRead/rowsWritten

rowsRead/rowsWritten

it's what I would expect. I will share it with the DO team to see if they find anything.

BurritoOP•2/6/25, 9:33 AM

Yeah the cursor numbers seem to be accurate, but dashboard numbers don't match and are way higher.

Llambrospetrou Looking at the code, I ran it and using the cursor `rowsRead/rowsWritten` it's w...

James•2/6/25, 4:12 PM

more importantly, which of these metrics are we being billed for, the ones feeding the dashboard incorrectly, or the cursor?

BurritoOP•2/6/25, 4:35 PM

The cursor numbers are what get billed, and dashboard numbers are supposed to match. It's unclear if the wrong numbers on dashboard are only for display, or if they are actually billed.

JJames more importantly, which of these metrics are we being billed for, the ones feedi...

lambrospetrou•2/6/25, 5:13 PM

BTW, just to make sure people are on the same page. Right now SQLite DOs, do NOT bill you for storage. It's part of the beta program. The team is aware of the discrepancy and will be fixed before actually charging folks.
https://developers.cloudflare.com/durable-objects/platform/pricing/#sqlite-storage-backend

During the initial beta, Storage API billing is not enabled for Durable Object classes using SQLite storage backend. SQLite-backed Durable Objects will incur charges for requests and duration. We plan to enable Storage API billing for Durable Objects using SQLite storage backend in the first half of 2025 after advance notice with the below pricing.

Cloudflare Docs

Pricing · Cloudflare Durable Objects docs

Durable Objects are only available on the Workers Paid plan, and are billed while the Durable Object is active (including the event context).

sup filistine•2/6/25, 7:00 PM

I am running cloudflare/puppeteeer as a client using a browser as a service outside of cloudflare (called Zenrows). The connection over

wss://

wss://

used to work just fine. We only used 1 Durable Object Rpc to do a 50s form filing process using the remote browser. This worked great.

Except as of last week, we are noticing this error like 80% of the time, when we do

puppeteer.connect

puppeteer.connect

Uncaught TypeError: Failed to establish the WebSocket connection: expected server to reply with HTTP status code 101 (switching protocols), but received 520 instead."

Uncaught TypeError: Failed to establish the WebSocket connection: expected server to reply with HTTP status code 101 (switching protocols), but received 520 instead."

Is anyone on the CF staff able to look into this on your end? I can provide my account ID and worker name.

(In talking with Zenrows, they haven't changed anything.)

sup filistine•2/6/25, 7:08 PM

X-posted here as well: workers-and-pages-helpSomething broken with establishing websocket connection with Browser as a service provider

If its better to respond to a post.

HHello, I’m Allie!Doesn’t that break the “One-Time” part of an OTP? Even if you immediately issue ...

iLearn•2/6/25, 9:29 PM

Correct and that is the reason i have designed DO per user (identified by mobile number).
BOTH - "one time" and "time based expiration, if unsed" are important in OTP (One Time Passcode" based authentication.

Ssathoro yeah better to use a DO w/ an alarm that deletes itself and also destroys the st...

iLearn•2/6/25, 9:35 PM

valid point.
but, wanted to save this step , as setting alarm + delete .. will eat 2 additional write operations and hence will add up bill (imagine, million OTP based auth happening per day and corresponding cost) .. which, other wise using Wokers-KV platform managed self-destruction will save these 2 steps and cost, as well ..

anyways .. that is more of business perspective .. and not tech side

Llambrospetrou I don't now what OTP codes meant by the user in that context, since time-based o...

iLearn•2/6/25, 9:43 PM

All points valid.
My Qs come from 2 dimensions .. Business + Tech ..
From Tech, ask is design need to be "race-free" + "as much as possible, platform should do house cleaning" .. DOs qualify "race-free" + Workers-KV (qualify house-cleaning)
From Business, ask is "do not burn hole for tasks which are not revenue generators"
(OTP based authentication is not revenue generator)

Anyways, thanks for kind response and sharing your understanding. I am working on design where DOs will be gate-keepers for access to ALL data stored in Workers-KV, and enjoy power of both world.
Shall post more Qs, as i progress and require clarifications.

IiLearn Just to be clear : fact#1 : "A request unit is defined as 4 KB of data read" (as...

iLearn•2/6/25, 9:47 PM

@lambrospetrou Can you please confirm answer for this Q. Thanks,

IiLearn All points valid. My Qs come from 2 dimensions .. Business + Tech .. From Tech...

lambrospetrou•2/6/25, 11:44 PM

I have a feeling you are overcomplicating things just to avoid doing the cleanup yourself with an alarm and 4 lines of code. You do realize DOs are cheaper 5x than Workers KV for writes, and only 2.5x more expensive for reads, so if you put DOs in the read path just to get the automatic delete from Workers KV, you get the worst of both worlds. And SQLite storage ops are even cheaper.

Anyway, your decision.

https://developers.cloudflare.com/durable-objects/platform/pricing/#storage-api-billing
https://developers.cloudflare.com/workers/platform/pricing/#workers-kv

IiLearn @lambrospetrou Can you please confirm answer for this Q. Thanks,

lambrospetrou•2/6/25, 11:49 PM

According to the docs:

Requests that hit the Durable Objects in-memory cache or that use the multi-key versions of get()/put()/delete() methods are billed the same as if they were a normal, individual request for each key.

https://developers.cloudflare.com/durable-objects/platform/pricing/#key-value-storage-backend

Llambrospetrou I have a feeling you are overcomplicating things just to avoid doing the cleanup...

iLearn•2/7/25, 12:39 AM

[ Following explanation are "general" discussion on DO and NOT in context of my OTP design, (though, principles will be evaluated/applied for OTP design as well, especially the race-condition offering) ]

I, respectfully, differ with statement "get the worst of both worlds".
Reason : "race-condition" management, offering by DO, is "price-less" (imho) . If i put DO as gate-keeper (in front of everything Workers-KV, R2), i make developer life much happier, much code management simpler (plz don't measure with the number of lines) with atleast blocking bugs popping up due to race-conditions while scaling the operations. Which otherwise, there will always a bulk of code to just manage race-conditions.

Also differ with statement "DOs are cheaper".
Reason : Using DOs, effectively, means you are "invoking" workers TWICE. One Master Worker which receives the request and the the DO stub call (which is effectively a worker call).

Also, DO-KV is charged per 4KB read, while Workers-KV is NOT charged based on size of data read/write.
The 5x and 2.5x cheap comparsion flies away with 5x and 2.5x size of data. Means, 20KB (in single write) data is break-even point and anything above this will make DO-KV writes "costlier". Similarly, for Read case.

Llambrospetrou According to the docs: > Requests that hit the Durable Objects in-memory cache o...

iLearn•2/7/25, 12:50 AM

Thanks for confirmation.

IiLearn [ Following explanation are "general" discussion on DO and NOT in context of my...

sathoro•2/7/25, 12:50 AM

you are just storing OTP, why would it be over 4kb of data?

import { DurableObject } from 'cloudflare:workers'; export class MyDurableObject extends DurableObject<Env> { constructor(ctx: DurableObjectState, env: Env) { super(ctx, env); ctx.blockConcurrencyWhile(async () => { await ctx.storage.transaction(async () => { const version = await ctx.storage.get('migrationVersion'); if (version === 1) return; ctx.storage.sql.exec(` CREATE TABLE Configs ( key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (key) ) STRICT; `); ctx.storage.sql.exec(`INSERT INTO Configs (key, value) VALUES (?, ?);`, 'foo', 'bar'); await ctx.storage.put('migrationVersion', 1); }); }); } async sayHello(key: string): Promise<string> { const { value } = this.ctx.storage.sql.exec(`SELECT value FROM Configs WHERE key = ?;`, key).one(); return `Hello, ${value}!`; } } export default { async fetch(request, env, ctx): Promise<Response> { const url = new URL(request.url); if (url.pathname !== '/hello-world') return new Response(null, { status: 404 }); let id: DurableObjectId = env.MY_DURABLE_OBJECT.idFromName('hello-world'); let stub = env.MY_DURABLE_OBJECT.get(id); let greeting = await stub.sayHello('foo'); return new Response(greeting); }, } satisfies ExportedHandler<Env>;

CREATE TABLE Configs ( key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (key) ) STRICT; CREATE TABLE Config1 ( key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (key) ) STRICT; CREATE TABLE Config2 ( key TEXT NOT NULL, value TEXT NOT NULL, PRIMARY KEY (key) ) STRICT; -- Config3, Config4, Config5

Is there a faster way to debug DO storage metrics than waiting tens of minutes for dashboard to upda

Similar Threads

Similar Threads

Similar Threads