has anyone else ran into "Durable Object storage is no longer accessible" in tests?

Ppeachneo has anyone else ran into "Durable Object storage is no longer accessible" in tes...

1

1984 Ford Laser•1/19/25, 5:12 AM

Are you trying to retrieve a key/value that was removed? Bit more context would be helpful

Llambrospetrou If you see my SQLSchemaMigrations helper, I don't need to read any rows, except ...

森

森@VerseLink (RW)•1/19/25, 5:13 AM

reading through the source code

森

森@VerseLink (RW)•1/19/25, 5:13 AM

await this.#_config.doStorage.put<number>(this.#_lastMigrationIDKeyName(), this.#_lastMigrationMonotonicId);

await this.#_config.doStorage.put<number>(this.#_lastMigrationIDKeyName(), this.#_lastMigrationMonotonicId);

await this.#_config.doStorage.put<number>(this.#_lastMigrationIDKeyName(), this.#_lastMigrationMonotonicId);

await this.#_config.doStorage.put<number>(this.#_lastMigrationIDKeyName(), this.#_lastMigrationMonotonicId);

森

森@VerseLink (RW)•1/19/25, 5:13 AM

shouldn't using transactionSync be enough?

森

森@VerseLink (RW)•1/19/25, 5:14 AM

since put / get is actually using special Sqlite table, and sqlite in DO is sync based, meaning there's no reason to ever await

森

森@VerseLink (RW)•1/19/25, 5:15 AM

https://developers.cloudflare.com/durable-objects/api/sql-storage/

Specifically for Durable Object classes with SQLite storage backend, KV operations which were previously asynchronous (for example, get, put, delete, deleteAll, list) are synchronous, even though they return promises. These methods will have completed their operations before they return the promise.

11984 Ford Laser Are you trying to retrieve a key/value that was removed? Bit more context would ...

P

peachneoOP•1/19/25, 5:18 AM

No... I'm running into a strange issue.

This works:

let ran = false;
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = false

let ran = false;
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = false

let ran = false;
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = false

let ran = false;
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = false

And this - which as far as I can tell is functionally identical.. does not work (throws the given error)

while (true) {
    let ran = await runDurableObjectAlarm(stub)
    if (!ran) {
      break
    };
}

while (true) {
    let ran = await runDurableObjectAlarm(stub)
    if (!ran) {
      break
    };
}

while (true) {
    let ran = await runDurableObjectAlarm(stub)
    if (!ran) {
      break
    };
}

while (true) {
    let ran = await runDurableObjectAlarm(stub)
    if (!ran) {
      break
    };
}

Ppeachneo No... I'm running into a strange issue. This works: ``` let ran = false; ran = ...

1

1984 Ford Laser•1/19/25, 5:19 AM

does

runDurableObjectAlarm()

runDurableObjectAlarm()

runDurableObjectAlarm()

runDurableObjectAlarm() return

true

true

or a similar value?

11984 Ford Laser does `runDurableObjectAlarm()` return `true` or a similar value?

P

peachneoOP•1/19/25, 5:19 AM

it returns a bool indicating whether the alarm executed or not

1

1984 Ford Laser•1/19/25, 5:21 AM

instead of

while (true)

while (true)

what about

while (ran)

while (ran)

? Declare

ran

ran

as true before the first run

11984 Ford Laser instead of `while (true)` what about `while (ran)`? Declare `ran` as true before...

P

peachneoOP•1/19/25, 5:23 AM

same result

1

1984 Ford Laser•1/19/25, 5:23 AM

Check what the alarm is returning to make sure its a bool?

11984 Ford Laser Check what the alarm is returning to make sure its a bool?

P

peachneoOP•1/19/25, 5:24 AM

yeah I've checked it's definitely a bool

1

1984 Ford Laser•1/19/25, 5:25 AM

What does the alarm do? Is the error in the alarm handler itself or this other code?

P

peachneoOP•1/19/25, 5:26 AM

it fails on a call to storage that the callback for the alarm calls

1

1984 Ford Laser•1/19/25, 5:29 AM

So then I assume the storage you're requesting isn't there? KV storage?

P

peachneoOP•1/19/25, 5:30 AM

durable object storage. Yeah I guess the storage isn't there... but the question is why lol

I mean how on earth does using a while loop that makes the exact same calls as the inlined non-loop version does result in a different outcome

P

peachneoOP•1/19/25, 5:31 AM

Smells like some weird race condition to me

Ppeachneo durable object storage. Yeah I guess the storage isn't there... but the question...

1

1984 Ford Laser•1/19/25, 5:33 AM

Very odd

P

peachneoOP•1/19/25, 6:26 AM

seems to be a timing issue, if I stick an arbitrary delay in the loop it passes

N

Niveth•1/19/25, 7:55 AM

What is the maximum time, I can set an alarm in durable object ?

NNiveth What is the maximum time, I can set an alarm in durable object ?

1

1984 Ford Laser•1/19/25, 9:00 AM

Maybe 03:14:07 UTC on 19 January 2038?

森森@VerseLink (RW)shouldn't using transactionSync be enough?

L

lambrospetrou•1/19/25, 9:00 AM

The KV API is async though, even though in SQlite DOs all storage operations are the same, so I am using

await

await

to be consistent with the API that you need to await a Promise. The underlying behavior doesn't change.
Someone reading the code and not seeing an

await

await

of a promise will be more confused, rather than having to read all the DO docs to figure out that it doesn't matter.

Llambrospetrou The KV API is async though, even though in SQlite DOs all storage operations are...

森

森@VerseLink (RW)•1/19/25, 9:03 AM

sounds good, yeah, I actually did notice that, I'm just thinking if the choice was intentional and if I have mistaken it or not

森

森@VerseLink (RW)•1/19/25, 9:03 AM

though, why doesn't DO expose a sync version of it under sql

森

森@VerseLink (RW)•1/19/25, 9:04 AM

i understand it's backwards compatiblity, but old DO can't migrate to sql backed DO though?

森森@VerseLink (RW)though, why doesn't DO expose a sync version of it under sql

L

lambrospetrou•1/19/25, 9:04 AM

because the KV API is the same for both types of DO, and in the non-SQLite one you actually have to await.

森

森@VerseLink (RW)•1/19/25, 9:05 AM

I'm actually writing similar migration tool myself (for internally stuff), would you say that creating a version table myself and use the sync api be better then KV API

森森@VerseLink (RW)I'm actually writing similar migration tool myself (for internally stuff), would...

L

lambrospetrou•1/19/25, 9:06 AM

There isn't better or worse. They are the same underlying storage. KV API goes to a special table that your SQL statements cannot modify, hence why I chose that. Otherwise you could mess up one of your actual migrations and delete your schema migration tracking table. Other than that separation, nothing else is different.

森

森@VerseLink (RW)•1/19/25, 9:07 AM

ah i see, thanks for the insight

森

森@VerseLink (RW)•1/19/25, 9:07 AM

I'm planning to keep track of the bookmarks when the migration happens (in case something bad happens, a rollback can be triggered immediately), so I would need an internal table to track that anyways

森森@VerseLink (RW)I'm planning to keep track of the bookmarks when the migration happens (in case ...

L

lambrospetrou•1/19/25, 9:09 AM

Choose the one that your code will be easier to read with and be correct

Both work.

Llambrospetrou Choose the one that your code will be easier to read with and be correct 🙂 Both...

森

森@VerseLink (RW)•1/19/25, 9:12 AM

oh also I'm also interested in using the export code that starbasedb offers, if the KV API Sql table is internal, does this mean we also need to export the KV data using list like original DO does?

森森@VerseLink (RW)oh also I'm also interested in using the export code that starbasedb offers, if ...

L

lambrospetrou•1/19/25, 9:13 AM

I haven't read all of the starbaseDB code, but if they only export SQL accessible data, I would assume that KV data is not included. Need to be tested though to be sure.

森

森@VerseLink (RW)•1/19/25, 9:14 AM

tldr it queries against sqlite_master

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

to get every table

森

森@VerseLink (RW)•1/19/25, 9:14 AM

not sure if the KV API's internal table is also tracked and can be queried against like this

森森@VerseLink (RW)tldr it queries against sqlite_master ```sql SELECT name FROM sqlite_master WHER...

L

lambrospetrou•1/19/25, 9:24 AM

You cannot access the KV table with SQL, just confirmed. There is also a test for it: https://github.com/cloudflare/workerd/blob/dbf8538fd806814acdfcbfec1f27c49b6f2f67b8/src/workerd/api/sql-test.js#L345

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

P

peachneoOP•1/19/25, 9:40 AM

do cloudflare devs have some internal library for testing DOs

Ppeachneo do cloudflare devs have some internal library for testing DOs

L

lambrospetrou•1/19/25, 9:42 AM

What do you mean? We use the public vitest integration too.

Llambrospetrou You cannot access the KV table with SQL, just confirmed. There is also a test fo...

森

森@VerseLink (RW)•1/19/25, 1:46 PM

:MeowHeartCloudflare:

Llambrospetrou because the KV API is the same for both types of DO, and in the non-SQLite one y...

B

Burrito•1/19/25, 4:26 PM

It'd be nice if SQLite DO can have a sync version of the KV API, even if it does exactly the same thing as the async one. I can get rid of the

blockConcurrencyWhile

blockConcurrencyWhile

when applying migration in constructor, and also I can put the entire migration into a

transactionSync

transactionSync

to prevent the rare event that migrations succeeded but writing migration number fails.

BBurrito It'd be nice if SQLite DO can have a sync version of the KV API, even if it does...

L

lambrospetrou•1/19/25, 4:35 PM

What would be the difference to use that instead of

transaction()

transaction()

?

blockConcurrencyWhile

blockConcurrencyWhile

has a different purpose than sync/async.

Llambrospetrou What would be the difference to use that instead of `transaction()`? `blockConcu...

B

Burrito•1/19/25, 4:51 PM

Say you have something like:

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

In DO constructor:

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

There are a few issues with this code:

Even if
```
storage.get
```
```
storage.get
```
/
```
storage.set
```
```
storage.set
```
's implementations are sync, because
```
applyMigration
```
```
applyMigration
```
is async,
```
.then
```
```
.then
```
puts continuation in the next event loop, so the log order will be
```
before
```
```
before
```
->
```
after
```
```
after
```
->
```
initialized
```
```
initialized
```
. I haven't tested it but unless DO has a different behavior to regular JS, that should be the case.
Because continuation goes on the next event loop, the DO is not fully initialized (
```
this.someState
```
```
this.someState
```
is still
```
undefined
```
```
undefined
```
) by the end of current event loop. I don't know if DO is able to process messages in the same event loop as the one doing the construction, but if that's true, then that's an issue.
Because initialization is in continuation, if you are using TS then you must null assert `someState!` because as far as TS can see, `this.someState` is not initialized when constructor exits.To fix some of these issues, you have to instead do:```tsclass MyDO extends DurableObject { someState!: number constructor(ctx: DurableObjectState, env: Env) { super(ctx, env) ctx.blockConcurrencyWhile(async () => { await applyMigrations(ctx.storage) this.someState = 42 }) }}```(This however still doesn't fix the last issue and you still have to `someState!`)If a sync KV API is exposed, this entire problem no longer exists.

森

森@VerseLink (RW)•1/19/25, 4:54 PM

I mean because KV API is actually sync, the blockConcurrencyWhile should complete execution kind of sync like, so it really isn't a issue

森

森@VerseLink (RW)•1/19/25, 4:54 PM

though the code looks a bit more uglier

B

Burrito•1/19/25, 4:56 PM

Yeah

blockConcurrencyWhile

blockConcurrencyWhile

is a workaround, but if we had an actual sync API, we wouldn't even need it.

BBurrito Say you have something like: ```ts async function applyMigrations(storage: Durab...

L

lambrospetrou•1/19/25, 4:56 PM

Your usecase seems to only need that due to Typescript typing. You can set a dummy value

this.someState = -1

this.someState = -1

before the call to

ctx.blockConcurrencyWhile(...)

ctx.blockConcurrencyWhile(...)

and then the property won't be considered optional.

Llambrospetrou Your usecase seems to only need that due to Typescript typing. You can set a dum...

B

Burrito•1/19/25, 4:58 PM

Sure, getting rid of that TS error is simple, but the point is that if we had a sync API we wouldn't need

blockConcurrencyWhile

blockConcurrencyWhile

at all, because

applyMigrations

applyMigrations

can just be sync.

B

Burrito•1/19/25, 5:01 PM

And because

storage.get

storage.get

/

storage.set

storage.set

is async, you also couldn't do:

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

Because just like the same problem above,

await

await

puts continuation in the next event loop, by which point

transactionSync

transactionSync

already exited.

BBurrito And because `storage.get`/`storage.set` is async, you also couldn't do: ```ts as...

L

lambrospetrou•1/19/25, 5:02 PM

That's why you also have

transaction(...)

transaction(...)

which is async. Anyway, we seem to be going in circles

has anyone else ran into "Durable Object storage is no longer accessible" in tests?

Similar Threads

Similar Threads

Similar Threads