durable object storage. Yeah I guess the storage isn't there... but the question is why lol I mean

durable object storage. Yeah I guess the storage isn't there... but the question is why lol

I mean how on earth does using a while loop that makes the exact same calls as the inlined non-loop version does result in a different outcome

peachneoOP•1/19/25, 5:31 AM

Smells like some weird race condition to me

Ppeachneo durable object storage. Yeah I guess the storage isn't there... but the question...

1984 Ford Laser•1/19/25, 5:33 AM

Very odd

peachneoOP•1/19/25, 6:26 AM

seems to be a timing issue, if I stick an arbitrary delay in the loop it passes

Niveth•1/19/25, 7:55 AM

What is the maximum time, I can set an alarm in durable object ?

NNiveth What is the maximum time, I can set an alarm in durable object ?

1984 Ford Laser•1/19/25, 9:00 AM

Maybe 03:14:07 UTC on 19 January 2038?

森森@VerseLink (RW)shouldn't using transactionSync be enough?

lambrospetrou•1/19/25, 9:00 AM

The KV API is async though, even though in SQlite DOs all storage operations are the same, so I am using

await

await

to be consistent with the API that you need to await a Promise. The underlying behavior doesn't change.
Someone reading the code and not seeing an

await

await

of a promise will be more confused, rather than having to read all the DO docs to figure out that it doesn't matter.

Llambrospetrou The KV API is async though, even though in SQlite DOs all storage operations are...

森

森@VerseLink (RW)•1/19/25, 9:03 AM

sounds good, yeah, I actually did notice that, I'm just thinking if the choice was intentional and if I have mistaken it or not

森

森@VerseLink (RW)•1/19/25, 9:03 AM

though, why doesn't DO expose a sync version of it under sql

森

森@VerseLink (RW)•1/19/25, 9:04 AM

i understand it's backwards compatiblity, but old DO can't migrate to sql backed DO though?

森森@VerseLink (RW)though, why doesn't DO expose a sync version of it under sql

lambrospetrou•1/19/25, 9:04 AM

because the KV API is the same for both types of DO, and in the non-SQLite one you actually have to await.

森

森@VerseLink (RW)•1/19/25, 9:05 AM

I'm actually writing similar migration tool myself (for internally stuff), would you say that creating a version table myself and use the sync api be better then KV API

森森@VerseLink (RW)I'm actually writing similar migration tool myself (for internally stuff), would...

lambrospetrou•1/19/25, 9:06 AM

There isn't better or worse. They are the same underlying storage. KV API goes to a special table that your SQL statements cannot modify, hence why I chose that. Otherwise you could mess up one of your actual migrations and delete your schema migration tracking table. Other than that separation, nothing else is different.

森

森@VerseLink (RW)•1/19/25, 9:07 AM

ah i see, thanks for the insight

森

森@VerseLink (RW)•1/19/25, 9:07 AM

I'm planning to keep track of the bookmarks when the migration happens (in case something bad happens, a rollback can be triggered immediately), so I would need an internal table to track that anyways

森森@VerseLink (RW)I'm planning to keep track of the bookmarks when the migration happens (in case ...

lambrospetrou•1/19/25, 9:09 AM

Choose the one that your code will be easier to read with and be correct

Both work.

Llambrospetrou Choose the one that your code will be easier to read with and be correct 🙂 Both...

森

森@VerseLink (RW)•1/19/25, 9:12 AM

oh also I'm also interested in using the export code that starbasedb offers, if the KV API Sql table is internal, does this mean we also need to export the KV data using list like original DO does?

森森@VerseLink (RW)oh also I'm also interested in using the export code that starbasedb offers, if ...

lambrospetrou•1/19/25, 9:13 AM

I haven't read all of the starbaseDB code, but if they only export SQL accessible data, I would assume that KV data is not included. Need to be tested though to be sure.

森

森@VerseLink (RW)•1/19/25, 9:14 AM

tldr it queries against sqlite_master

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

to get every table

森

森@VerseLink (RW)•1/19/25, 9:14 AM

not sure if the KV API's internal table is also tracked and can be queried against like this

森森@VerseLink (RW)tldr it queries against sqlite_master ```sql SELECT name FROM sqlite_master WHER...

lambrospetrou•1/19/25, 9:24 AM

You cannot access the KV table with SQL, just confirmed. There is also a test for it: https://github.com/cloudflare/workerd/blob/dbf8538fd806814acdfcbfec1f27c49b6f2f67b8/src/workerd/api/sql-test.js#L345

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

peachneoOP•1/19/25, 9:40 AM

do cloudflare devs have some internal library for testing DOs

Ppeachneo do cloudflare devs have some internal library for testing DOs

lambrospetrou•1/19/25, 9:42 AM

What do you mean? We use the public vitest integration too.

Llambrospetrou You cannot access the KV table with SQL, just confirmed. There is also a test fo...

森

森@VerseLink (RW)•1/19/25, 1:46 PM

:MeowHeartCloudflare:

Llambrospetrou because the KV API is the same for both types of DO, and in the non-SQLite one y...

Burrito•1/19/25, 4:26 PM

It'd be nice if SQLite DO can have a sync version of the KV API, even if it does exactly the same thing as the async one. I can get rid of the

blockConcurrencyWhile

blockConcurrencyWhile

when applying migration in constructor, and also I can put the entire migration into a

transactionSync

transactionSync

to prevent the rare event that migrations succeeded but writing migration number fails.

BBurrito It'd be nice if SQLite DO can have a sync version of the KV API, even if it does...

lambrospetrou•1/19/25, 4:35 PM

What would be the difference to use that instead of

transaction()

transaction()

blockConcurrencyWhile

blockConcurrencyWhile

has a different purpose than sync/async.

Llambrospetrou What would be the difference to use that instead of `transaction()`? `blockConcu...

Burrito•1/19/25, 4:51 PM

Say you have something like:

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

In DO constructor:

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

There are a few issues with this code:

Even if
```
storage.get
```
```
storage.get
```
/
```
storage.set
```
```
storage.set
```
's implementations are sync, because
```
applyMigration
```
```
applyMigration
```
is async,
```
.then
```
```
.then
```
puts continuation in the next event loop, so the log order will be
```
before
```
```
before
```
->
```
after
```
```
after
```
->
```
initialized
```
```
initialized
```
. I haven't tested it but unless DO has a different behavior to regular JS, that should be the case.
Because continuation goes on the next event loop, the DO is not fully initialized (
```
this.someState
```
```
this.someState
```
is still
```
undefined
```
```
undefined
```
) by the end of current event loop. I don't know if DO is able to process messages in the same event loop as the one doing the construction, but if that's true, then that's an issue.
Because initialization is in continuation, if you are using TS then you must null assert `someState!` because as far as TS can see, `this.someState` is not initialized when constructor exits.To fix some of these issues, you have to instead do:```tsclass MyDO extends DurableObject { someState!: number constructor(ctx: DurableObjectState, env: Env) { super(ctx, env) ctx.blockConcurrencyWhile(async () => { await applyMigrations(ctx.storage) this.someState = 42 }) }}```(This however still doesn't fix the last issue and you still have to `someState!`)If a sync KV API is exposed, this entire problem no longer exists.

森

森@VerseLink (RW)•1/19/25, 4:54 PM

I mean because KV API is actually sync, the blockConcurrencyWhile should complete execution kind of sync like, so it really isn't a issue

森

森@VerseLink (RW)•1/19/25, 4:54 PM

though the code looks a bit more uglier

Burrito•1/19/25, 4:56 PM

Yeah

blockConcurrencyWhile

blockConcurrencyWhile

is a workaround, but if we had an actual sync API, we wouldn't even need it.

BBurrito Say you have something like: ```ts async function applyMigrations(storage: Durab...

lambrospetrou•1/19/25, 4:56 PM

Your usecase seems to only need that due to Typescript typing. You can set a dummy value

this.someState = -1

this.someState = -1

before the call to

ctx.blockConcurrencyWhile(...)

ctx.blockConcurrencyWhile(...)

and then the property won't be considered optional.

Llambrospetrou Your usecase seems to only need that due to Typescript typing. You can set a dum...

Burrito•1/19/25, 4:58 PM

Sure, getting rid of that TS error is simple, but the point is that if we had a sync API we wouldn't need

blockConcurrencyWhile

blockConcurrencyWhile

at all, because

applyMigrations

applyMigrations

can just be sync.

Burrito•1/19/25, 5:01 PM

And because

storage.get

storage.get

storage.set

storage.set

is async, you also couldn't do:

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

Because just like the same problem above,

await

await

puts continuation in the next event loop, by which point

transactionSync

transactionSync

already exited.

BBurrito And because `storage.get`/`storage.set` is async, you also couldn't do: ```ts as...

lambrospetrou•1/19/25, 5:02 PM

That's why you also have

transaction(...)

transaction(...)

which is async. Anyway, we seem to be going in circles

Burrito•1/19/25, 5:02 PM

Can you use

transaction

transaction

with SQLite?

BBurrito Can you use `transaction` with SQLite?

lambrospetrou•1/19/25, 5:03 PM

Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction

When using the SQLite-backed storage engine, the txn object is obsolete. Any storage operations performed directly on the ctx.storage object, including SQL queries using ctx.storage.sql.exec(), will be considered part of the transaction.

You can use both APIs, KV and SQL, inside an SQLite DO.

Burrito•1/19/25, 5:04 PM

Ah, for some reason I was under the impression that I must use

transactionSync

transactionSync

with SQLite.

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

森

森@VerseLink (RW)•1/19/25, 5:04 PM

btw does trx.rollback() work with SQL storage backend?

Burrito•1/19/25, 5:05 PM

Cool, that gets rid of one part of the pain points then.

森

森@VerseLink (RW)•1/19/25, 5:05 PM

or do we just throw exceptions

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

Hello, I’m Allie!•1/19/25, 6:23 PM

Can you do transactions across different I/O events?

HHello, I’m Allie!Can you do transactions across different I/O events?

lambrospetrou•1/19/25, 6:40 PM

What do you refer to as IO event?

森

森@VerseLink (RW)•1/19/25, 6:41 PM

btw it says stream api is only available in request context, does DOs RPC method consider a request context?

森

森@VerseLink (RW)•1/19/25, 7:46 PM

huh does DO have in-memory size limit?

Llambrospetrou What do you refer to as IO event?

Hello, I’m Allie!•1/19/25, 8:01 PM

Like, can I start a transaction within a fetch handler, then close it out within an RPC?

Hello, I’m Allie!•1/19/25, 8:01 PM

Or another combo of RPC/fetch/WebSocketMessage, etc

森森@VerseLink (RW)huh does DO have in-memory size limit?

Hello, I’m Allie!•1/19/25, 8:08 PM

Yes, same as a Worker, 128 MB

HHello, I’m Allie!Yes, same as a Worker, 128 MB

森

森@VerseLink (RW)•1/19/25, 8:09 PM

oh i thought they meant DO will always occupy 128MB, didn't know that is also the limit

Skrrrrrrrrr•1/19/25, 8:48 PM

is there some issues with the Turkey server?

HHello, I’m Allie!Like, can I start a transaction within a fetch handler, then close it out within...

lambrospetrou•1/19/25, 8:49 PM

The way you do an async transaction now is through

transaction(...)

transaction(...)

, so what you suggest could be done by starting one in one fetch/rpc call, but without awaiting the returned promise and just storing it in-memory. Then another request can await for it when needed.
This should work, but again, consistency is up to you to make sure what you do inside the transaction call is OK.
For example, if you try to iterate a cursor across IO context boundaries, I think it won't work (well-defined).

Llambrospetrou The way you do an async transaction now is through `transaction(...)`, so what y...

Hello, I’m Allie!•1/19/25, 8:51 PM

But you wouldn't be able to close out the transaction with context from the second event, right?

durable object storage. Yeah I guess the storage isn't there... but the question is why lol I mean

Similar Threads

Similar Threads

Similar Threads