Cloudflare Developers•11mo ago

You mean instead of adding the 2 lines yourself, or is there something else in this?

SnowBall•2/4/25, 10:07 AM

SnowBall•2/4/25, 10:08 AM

有中国的哥们吗

Llambrospetrou You mean instead of adding the 2 lines yourself, or is there something else in t...

1984 Ford Laser•2/4/25, 10:22 AM

I mean when you put it like that...

Nnil wrote a util to help with maximum 6 concurrent network requests... has anyone do...

sathoro•2/4/25, 11:05 AM

why do you need to do this? it will queue the requests automatically

Nnil Might have been a wrangler dev issue but I ran into a resource limit: "Response ...

Isaac McFadyen•2/4/25, 7:16 PM

Yeah I've gotten this as well in production

iLearn•2/4/25, 8:43 PM

Are the options "options: { expiration?: number, expirationTtl?: number, metadata?: object }" available for Workers-KV available for DO-KV storage ?

Checked the link "https://developers.cloudflare.com/durable-objects/api/storage-api/#supported-options-1" and could not find the aforesaid options and hence wanted to confirm.

Cloudflare Docs

Durable Object Storage · Cloudflare Durable Objects docs

The Durable Object Storage API allows Durable Objects to access transactional and strongly consistent storage. A Durable Object's attached storage is private to its unique instance and cannot be accessed by other objects.

IiLearn Are the options "options: { expiration?: number, expirationTtl?: number, metadat...

1984 Ford Laser•2/4/25, 8:47 PM

Nah I don't think so, Workers KV and DO KV are technically different. Like Workers KV will show a list()list() as the keys, whereas DO KV returns the k:v pairs

11984 Ford Laser Nah I don't think so, Workers KV and DO KV are technically different. Like Worke...

iLearn•2/4/25, 8:50 PM

Thanks for sharing your understanding.

iLearn•2/4/25, 8:55 PM

Since, expirationTtl (option in Workers-KV) is not an option for DO-KV, what are the ways to achieve the same functionality where the Key need to be deleted after, say 10 minutes ?

iLearn•2/4/25, 8:56 PM

Since, expirationTtl (option in Workers-KV) is not an option for DO-KV, what are the ways to achieve the same functionality where the Key need to be deleted after, say 10 minutes ?

Original message was deleted

iLearn•2/4/25, 9:03 PM

Thanks for sharing your understanding.
now i am in catch-22 situation .. wanted to use DO-KV to use ready-made infra for race-condition-avoidance and all goodies of Workers-KV .. but, now with the missing option of expirationTtl .. considering to go back to Workers-KV ..

choice is either manage race-condition or manage auto delete of key using expirationTtl

ssandler•2/4/25, 9:44 PM

Has anyone run into troubles using otel-cf-workers for tracing with the latest durable objects sdk? workers-observability

iLearn•2/5/25, 6:01 AM

QQ regarding pricing behaviour of DO-KV :
While exercising Option "get(keys)", where keys will be an array .. assume 10 Keys are requested and 10 K-V are returned (assume total size of complete read is less than 4KB) .. my cost will 1 Read request units (and NOT 10 Keys x Price per Key Read, as behaved by Workers-KV) ? Please confirm.

IiLearn QQ regarding pricing behaviour of DO-KV : While exercising Option "get(keys)", w...

1984 Ford Laser•2/5/25, 7:57 AM

Nah get()get() is a storage operation per key, list()list() is per 4KB. If a single key in a get is over 4KB that will obvs be multiple ops

1984 Ford Laser•2/5/25, 7:59 AM

Learnt this one the hard way after I'd already determined a KV schema I liked. Will go back and optimise later, taking advantage of namespacing helps here. For example, if you had a number fields that belonged to a 'config' object, you might store each key as 'config-{key_name}' so then you're able to either pull the key itself, or all keys under 'config-' using list which is cheaper for reads

IiLearn Thanks for sharing your understanding. now i am in catch-22 situation .. wanted ...

lambrospetrouOP•2/5/25, 11:04 AM

Handling expiration after a TTL is quite trivial though if you need the single threaded coordination of DOs.
Attach a timestamp + a TTL on your values, and when reading them just check them against the current timestamp. If it's expired assume it didn't exist, and optionally delete it for cleanup. That's it.

Hard@Work•2/5/25, 11:06 AM

Hey @lambrospetrou, I forget, is there a way to push arbitrary objects into SQLite? I know that the backend can do it for SQLiteKV, but is there a way to do it when directly querying tables?

11984 Ford Laser Nah `get()` is a storage operation per key, `list()` is per 4KB. If a single key...

lambrospetrouOP•2/5/25, 11:06 AM

Check also the new SQLite DOs, since they could be much cheaper in terms of storage operations, depending on your access patterns: https://developers.cloudflare.com/durable-objects/platform/pricing/#sqlite-storage-backend

HHard@Work Hey @lambrospetrou, I forget, is there a way to push arbitrary objects into SQLi...

lambrospetrouOP•2/5/25, 11:07 AM

What do you mean by push arbitrary objects?

Llambrospetrou What do you mean by push arbitrary objects?

Hard@Work•2/5/25, 11:07 AM

Like how DOKV can store anything that can be structuredClonestructuredCloned

HHard@Work Like how DOKV can store anything that can be `structuredClone`d

lambrospetrouOP•2/5/25, 11:09 AM

SQLite supports bytes, so you could store anything if you convert it to bytes, or just serialize them on your own and store it as a blob or text. The conversion is not automatic though, you do it when you

INSERT

INSERT

rows.

Llambrospetrou SQLite supports bytes, so you could store anything if you convert it to bytes, o...

Hard@Work•2/5/25, 11:10 AM

That would require access to the V8 Serializer(or a custom one) within the runtime, which doesn't exist yet, right?

HHard@Work That would require access to the V8 Serializer(or a custom one) within the runti...

lambrospetrouOP•2/5/25, 11:13 AM

I am not sure if the Workers runtime provides that serializer for users, to be honest.

geg•2/5/25, 12:41 PM

I asked about it before

mentioned here:
durable-objects

Llambrospetrou SQLite supports bytes, so you could store anything if you convert it to bytes, o...

Hard@Work•2/5/25, 12:43 PM

Actually, Kenton has a good point about giving users access to serialisers directly(being that upgradability is nigh-on impossible if the user has to serialize data themselves). I'm not too familiar with SQLite Extensions/Plugins, but would it be possible to somehow add a serialiser to the SQLite API, so it wouldn't require direct access?

Original message was deleted

Hello, I’m Allie!•2/5/25, 12:53 PM

Not necessarily? I was thinking that it would be some kind of extension where, if it recognizes an “object” column, then all modifications pass through the serializer, so you can’t ever directly touch the bytes

Hard@Work•2/5/25, 12:55 PM

Yeah idk

Original message was deleted

lambrospetrouOP•2/5/25, 2:52 PM

Keep in mind this is your database, in your application, and your business logic is really tied to it. I personally wouldn't depend on some abstract serialization for putting data in my SQL database. For the KV API it's fine since that's abstracted from you higher up, you "put" an object and you "get" that object back. You don't have direct access to the underlying bytes.

With SQLite, you do since you say what goes into each row. It's your database tables. In that case I would favor using serialization that I control. You can migrate from one to another if you change your mind, or want better compression in size vs time, etc.

Burrito•2/5/25, 4:12 PM

I'm using SQLite DO, one request is counting for 236 row reads, but with logging I've confirmed that there are only 5 SQL queries each with 1 row read. What could the rest be from? Does using the KV API count for more row reads?

BBurrito I'm using SQLite DO, one request is counting for 236 row reads, but with logging...

lambrospetrouOP•2/5/25, 4:15 PM

You say 5 SQL queries, but then you mention the KV API. What are you doing within each request?

If you are using SQL, then keep in mind any select you do might access your indices as well, and if you don't have indices a full scan is done reading all your rows.

Llambrospetrou You say 5 SQL queries, but then you mention the KV API. What are you doing withi...

Burrito•2/5/25, 4:18 PM

In DO constructor I use the KV API for migration, this should be 1 get operation there to check if the migration is up to date.
Code afterwards uses SQL API only, exposed via a wrapper:

const cursor = sqlStorage.exec(query, ...values)
const results = [...cursor]

if (logSql) {
    console.log({
        query,
        read: cursor.rowsRead,
        written: cursor.rowsWritten,
    })
}

return results

const cursor = sqlStorage.exec(query, ...values)
const results = [...cursor]

if (logSql) {
    console.log({
        query,
        read: cursor.rowsRead,
        written: cursor.rowsWritten,
    })
}

return results

Burrito•2/5/25, 4:25 PM

The request results in 236 row reads in DO metrics, but worker logs only show 5 SQL queries with 1 row read each. The only thing unaccounted for is that one KV get at the start, which is why I'm asking if KV API has a different way of counting.

11984 Ford Laser Learnt this one the hard way after I'd already determined a KV schema I liked. W...

iLearn•2/5/25, 4:40 PM

Brilliant Answer. Effort (and Money

) saver for Architects.
I was about to fall in the same trap as yours, but somehow thought to reach community to validate my schema and pricing. And bingo, your answer saved

@lambrospetrou @Hard@Work | R2 (Please pardon for tagging, if i am breaking community rules): In case you are Cloudflare employees, may you please update the documentation with the above answer (as an example) in "Pricing" for DO-KV storage. This will be "money"-saver for many, as the Pricing page has no such "explicit" example showing the clear "distinction" in pricing approach for "list" and "get" (though for list(), it is very clearly mentioned, but NOT for get() / put() ). Thanks.
Pricing page link : https://developers.cloudflare.com/durable-objects/platform/pricing/#key-value-storage-backend

Llambrospetrou Handling expiration after a TTL is quite trivial though if you need the single t...

iLearn•2/5/25, 4:50 PM

Appreciate the suggestion.
expirationTTL, imho, is of more value as ready-made-infra for "cleaning" the stale data (store and forget). There are many keys, which one "wants to forget" after creating.
example use-case : 2-minute expiration OTPs. i shall be relieved that i can keep on storing multiple such OTPs and they will be auto-cleaned up by infra. (NOTE : validation of OTP based on expiration time can be acheived in same way as you suggested in worker and hence, validation is not much of a concern). Concern will be a user creates multiple OTPs and "DOES-NOT" use them, in this scenario, expirationTTL keep my KV-store cleaning up and make developer life worry-free.

BBurrito The request results in 236 row reads in DO metrics, but worker logs only show 5 ...

James•2/5/25, 4:54 PM

I find it easier to debug row counting / query optimization when I log the return values of the rows read / written per query in real-time, rather than deduce from less granular logs.

JJames I find it easier to debug row counting / query optimization when I log the retur...

Burrito•2/5/25, 4:57 PM

Well

cursor.rowsRead

cursor.rowsRead

should be accurate, the amount of returned rows isn't a good representation of amount of rows read because query can read more than it returns.

Burrito•2/5/25, 4:57 PM

But the query in question is just look up row by primary key, so 1 row read is accurate regardless.

IiLearn Brilliant Answer. Effort (and Money 🙂 ) saver for Architects. I was about to f...

lambrospetrouOP•2/5/25, 5:12 PM

What exactly is it not mentioned in the pricing docs?

A request unit is defined as 4 KB of data read or written. A request that writes or reads more than 4 KB will consume multiple units, for example, a 9 KB write will consume 3 write request units.

This is an example on the pricing page explaining how reads (e.g. gets) work. List is the same, but it needs clarification to make sure people know that it's across all items examined/returned.

Llambrospetrou What exactly is it not mentioned in the pricing docs? > A request unit is def...

iLearn•2/5/25, 5:18 PM

Just to be clear :
fact#1 : "A request unit is defined as 4 KB of data read" (as mentioned in Pricing page)

My Q#1 : Assume, "keys" is an array of 10-Keys, if i make get(keys) (single operation), and cumulative size of all 10-K-V returned values is less than 4KB AND size of each K-V is also less than 4KB .. then what will be the pricing ?
Answer Option 1 : 10 x Price of each RRU
Answer Option 2 : 1 x Price of each RRU

Link of usage of get(keys) .. NOTE : keys (in PLURAL) :
[ get(keys) is NOT available for Workers-KV, it is ONLY for DO-KV ]
https://developers.cloudflare.com/durable-objects/api/storage-api/#get

Cloudflare Docs

Durable Object Storage · Cloudflare Durable Objects docs

Burrito•2/5/25, 6:03 PM

Is there a faster way to debug DO storage metrics than waiting tens of minutes for dashboard to update? Is there a way to do it locally?

BBurrito Is there a faster way to debug DO storage metrics than waiting tens of minutes f...

Hard@Work•2/5/25, 6:05 PM

I don't think so? IIRC the dashboard is powered by GraphQL, which you can query directly, however it would still have delays

Burrito•2/5/25, 6:06 PM

Well that's problematic.

BBurrito Well that's problematic.

Hard@Work•2/5/25, 6:07 PM

It's a difficult thing because while seeing storage metrics for a single DO might be easy, seeing aggregated metrics for millions/billions of DOs fast is very difficult

Burrito•2/5/25, 6:08 PM

Yeah that's understandable, but I would've thought there's a way to do it locally, since the runtime is now open source and all that.

BBurrito The request results in 236 row reads in DO metrics, but worker logs only show 5 ...

Burrito•2/5/25, 6:09 PM

I'm trying to debug this issue and it's really not helping that I need to wait a long time before I can observe what had happened.

BBurrito Yeah that's understandable, but I would've thought there's a way to do it locall...

Hard@Work•2/5/25, 6:09 PM

Ooh, ok. I'm not entirely sure, is the DO storage engine actually open source?

Hard@Work•2/5/25, 6:10 PM

I don't remember if it uses a stub within Miniflare

Burrito•2/5/25, 6:10 PM

Ah not sure then.

You mean instead of adding the 2 lines yourself, or is there something else in this?

Similar Threads

Similar Threads

Similar Threads