It'd be nice if SQLite DO can have a sync version of the KV API, even if it does exactly the same th

BBurrito It'd be nice if SQLite DO can have a sync version of the KV API, even if it does...

L

What would be the difference to use that instead of

transaction()

transaction()

transaction()

transaction()?

blockConcurrencyWhile

blockConcurrencyWhile

has a different purpose than sync/async.

Llambrospetrou What would be the difference to use that instead of `transaction()`? `blockConcu...

B

BurritoOP•1/19/25, 4:51 PM

Say you have something like:

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

In DO constructor:

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

There are a few issues with this code:

Even if
```
storage.get
```
storage.get
```
storage.get
```
storage.get
/
```
storage.set
```
storage.set
```
storage.set
```
storage.set
's implementations are sync, because
```
applyMigration
```
applyMigration
```
applyMigration
```
applyMigration
is async,
```
.then
```
.then
```
.then
```
.then
puts continuation in the next event loop, so the log order will be
```
before
```
before
```
before
```
before
->
```
after
```
after
```
after
```
after
->
```
initialized
```
initialized
```
initialized
```
initialized
. I haven't tested it but unless DO has a different behavior to regular JS, that should be the case.
Because continuation goes on the next event loop, the DO is not fully initialized (
```
this.someState
```
this.someState
```
this.someState
```
this.someState
is still
```
undefined
```
```
undefined
```
) by the end of current event loop. I don't know if DO is able to process messages in the same event loop as the one doing the construction, but if that's true, then that's an issue.
Because initialization is in continuation, if you are using TS then you must null assert `someState!` because as far as TS can see, `this.someState` is not initialized when constructor exits.To fix some of these issues, you have to instead do:```tsclass MyDO extends DurableObject { someState!: number constructor(ctx: DurableObjectState, env: Env) { super(ctx, env) ctx.blockConcurrencyWhile(async () => { await applyMigrations(ctx.storage) this.someState = 42 }) }}```(This however still doesn't fix the last issue and you still have to `someState!`)If a sync KV API is exposed, this entire problem no longer exists.

森

森@VerseLink (RW)•1/19/25, 4:54 PM

I mean because KV API is actually sync, the blockConcurrencyWhile should complete execution kind of sync like, so it really isn't a issue

森

森@VerseLink (RW)•1/19/25, 4:54 PM

though the code looks a bit more uglier

B

BurritoOP•1/19/25, 4:56 PM

Yeah

blockConcurrencyWhile

blockConcurrencyWhile

is a workaround, but if we had an actual sync API, we wouldn't even need it.

BBurrito Say you have something like: ```ts async function applyMigrations(storage: Durab...

L

lambrospetrou•1/19/25, 4:56 PM

Your usecase seems to only need that due to Typescript typing. You can set a dummy value

this.someState = -1

this.someState = -1

this.someState = -1

this.someState = -1 before the call to

ctx.blockConcurrencyWhile(...)

ctx.blockConcurrencyWhile(...)

ctx.blockConcurrencyWhile(...)

ctx.blockConcurrencyWhile(...) and then the property won't be considered optional.

Llambrospetrou Your usecase seems to only need that due to Typescript typing. You can set a dum...

B

BurritoOP•1/19/25, 4:58 PM

Sure, getting rid of that TS error is simple, but the point is that if we had a sync API we wouldn't need

blockConcurrencyWhile

blockConcurrencyWhile

at all, because

applyMigrations

applyMigrations

applyMigrations

applyMigrations can just be sync.

B

BurritoOP•1/19/25, 5:01 PM

And because

storage.get

storage.get

storage.get

storage.get/

storage.set

storage.set

storage.set

storage.set is async, you also couldn't do:

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

Because just like the same problem above,

await

await

await

await puts continuation in the next event loop, by which point

transactionSync

transactionSync

already exited.

BBurrito And because `storage.get`/`storage.set` is async, you also couldn't do: ```ts as...

L

lambrospetrou•1/19/25, 5:02 PM

That's why you also have

transaction(...)

transaction(...)

transaction(...)

transaction(...) which is async. Anyway, we seem to be going in circles

B

BurritoOP•1/19/25, 5:02 PM

Can you use

transaction

transaction

transaction

transaction with SQLite?

BBurrito Can you use `transaction` with SQLite?

L

lambrospetrou•1/19/25, 5:03 PM

Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction

When using the SQLite-backed storage engine, the txn object is obsolete. Any storage operations performed directly on the ctx.storage object, including SQL queries using ctx.storage.sql.exec(), will be considered part of the transaction.

You can use both APIs, KV and SQL, inside an SQLite DO.

B

BurritoOP•1/19/25, 5:04 PM

Ah, for some reason I was under the impression that I must use

transactionSync

transactionSync

with SQLite.

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

森

森@VerseLink (RW)•1/19/25, 5:04 PM

btw does trx.rollback() work with SQL storage backend?

B

BurritoOP•1/19/25, 5:05 PM

Cool, that gets rid of one part of the pain points then.

森

森@VerseLink (RW)•1/19/25, 5:05 PM

or do we just throw exceptions

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

H

Hello, I’m Allie!•1/19/25, 6:23 PM

Can you do transactions across different I/O events?

HHello, I’m Allie!Can you do transactions across different I/O events?

L

lambrospetrou•1/19/25, 6:40 PM

What do you refer to as IO event?

森

森@VerseLink (RW)•1/19/25, 6:41 PM

btw it says stream api is only available in request context, does DOs RPC method consider a request context?

森

森@VerseLink (RW)•1/19/25, 7:46 PM

huh does DO have in-memory size limit?

Llambrospetrou What do you refer to as IO event?

H

Hello, I’m Allie!•1/19/25, 8:01 PM

Like, can I start a transaction within a fetch handler, then close it out within an RPC?

H

Hello, I’m Allie!•1/19/25, 8:01 PM

Or another combo of RPC/fetch/WebSocketMessage, etc

森森@VerseLink (RW)huh does DO have in-memory size limit?

H

Hello, I’m Allie!•1/19/25, 8:08 PM

Yes, same as a Worker, 128 MB

HHello, I’m Allie!Yes, same as a Worker, 128 MB

森

森@VerseLink (RW)•1/19/25, 8:09 PM

oh i thought they meant DO will always occupy 128MB, didn't know that is also the limit

S

Skrrrrrrrrr•1/19/25, 8:48 PM

is there some issues with the Turkey server?

HHello, I’m Allie!Like, can I start a transaction within a fetch handler, then close it out within...

L

lambrospetrou•1/19/25, 8:49 PM

The way you do an async transaction now is through

transaction(...)

transaction(...)

transaction(...)

transaction(...), so what you suggest could be done by starting one in one fetch/rpc call, but without awaiting the returned promise and just storing it in-memory. Then another request can await for it when needed.
This should work, but again, consistency is up to you to make sure what you do inside the transaction call is OK.
For example, if you try to iterate a cursor across IO context boundaries, I think it won't work (well-defined).

Llambrospetrou The way you do an async transaction now is through `transaction(...)`, so what y...

H

Hello, I’m Allie!•1/19/25, 8:51 PM

But you wouldn't be able to close out the transaction with context from the second event, right?

HHello, I’m Allie!But you wouldn't be able to close out the transaction with context from the seco...

L

lambrospetrou•1/19/25, 8:52 PM

What do you mean "close" the transaction? The transaction is a callback that you pass to the

transaction()

transaction()

transaction()

transaction() method.

Llambrospetrou What do you mean "close" the transaction? The transaction is a callback that you...

森

森@VerseLink (RW)•1/19/25, 8:59 PM

I guess they meant creating a promise in the transaction and store the resolve function elsewhere (so the transaction keeps waiting) until it is the resolve function is resolved on a different / second event.
(But wouldn't that be a deadlock?)

森

森@VerseLink (RW)•1/19/25, 9:02 PM

@lambrospetrou hey, quick question, is it possible to create a Mock DO using vitest to test for functionality that relies on DO's SQL storage?
Mock DO as in, a dummy DO that only exists to provide it's SQL storage logic

森森@VerseLink (RW)@lambrospetrou hey, quick question, is it possible to create a Mock DO using vit...

L

lambrospetrou•1/19/25, 9:04 PM

https://github.com/cloudflare/workers-sdk/blob/main/fixtures/vitest-pool-workers-examples/durable-objects/test/direct-access.test.ts

Llambrospetrou What do you mean "close" the transaction? The transaction is a callback that you...

H

Hello, I’m Allie!•1/19/25, 9:06 PM

For example, if within the first fetch/rpc I do

// Using KV for simplicity
const a = txn.get("foo")

// Using KV for simplicity
const a = txn.get("foo")

// Using KV for simplicity
const a = txn.get("foo")

// Using KV for simplicity
const a = txn.get("foo")

I wouldn't be able to do this in the second fetch/rpc

txn.put("bar", a + "yo");

txn.put("bar", a + "yo");

txn.put("bar", a + "yo");

txn.put("bar", a + "yo");

H

Hello, I’m Allie!•1/19/25, 9:07 PM

Actually, this example doesn't really display a need for a transaction, but I meant more that you can't start a transaction within one I/O context, then complete it within another with data received in that context

HHello, I’m Allie!For example, if within the first fetch/rpc I do ```ts // Using KV for simplicity...

L

lambrospetrou•1/19/25, 9:12 PM

I still don't see what usecase would be good for blocking your transactions across requests. You just increase the likelyhood of having them fail due to conflicts at best. If I remember correctly, DOs have their own context, so in theory you can do what you say.

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

I "think" this should be doable, but I didn't confirm/test it. I would never do it though

Llambrospetrou https://github.com/cloudflare/workers-sdk/blob/main/fixtures/vitest-pool-workers...

森

森@VerseLink (RW)•1/19/25, 9:16 PM

hmm, my question is more like is it possible to have a dummy / mock DO class in the test folder, instead of in src folder that interferes with the exports of my actual app.

If not, would you suggest instead to create a different isolated package (for example in a monorepo)?

森森@VerseLink (RW)hmm, my question is more like is it possible to have a dummy / mock DO class in ...

L

lambrospetrou•1/19/25, 9:17 PM

I don't understand what you are asking. You can create your own worker and its Durable Objects bind to it with miniflare if you want and use that.

Llambrospetrou I don't understand what you are asking. You can create your own worker and its D...

森

森@VerseLink (RW)•1/19/25, 9:30 PM

Wait, actually thank you! Your hint of "miniflare" and that github example link really helped out big time!

森

森@VerseLink (RW)•1/19/25, 10:05 PM

hmm now i'm getting

Error: Durable Object test helpers can only be used with stubs pointing to objects defined within the same worker.

森

森@VerseLink (RW)•1/19/25, 10:07 PM

here's my miniflare config

miniflare: {
  workers: [
    {
      name: "dummy-do",
      modules: true,
      scriptPath: "./test/database/mo.ts"
    }
  ],
  durableObjects: {
    "MockSqlBackupDO": { className: "DummyDO", scriptName: "dummy-do", useSQLite: true }
  }
}

miniflare: {
  workers: [
    {
      name: "dummy-do",
      modules: true,
      scriptPath: "./test/database/mo.ts"
    }
  ],
  durableObjects: {
    "MockSqlBackupDO": { className: "DummyDO", scriptName: "dummy-do", useSQLite: true }
  }
}

miniflare: {
  workers: [
    {
      name: "dummy-do",
      modules: true,
      scriptPath: "./test/database/mo.ts"
    }
  ],
  durableObjects: {
    "MockSqlBackupDO": { className: "DummyDO", scriptName: "dummy-do", useSQLite: true }
  }
}

miniflare: {
  workers: [
    {
      name: "dummy-do",
      modules: true,
      scriptPath: "./test/database/mo.ts"
    }
  ],
  durableObjects: {
    "MockSqlBackupDO": { className: "DummyDO", scriptName: "dummy-do", useSQLite: true }
  }
}

E

Eric•1/20/25, 1:54 AM

Question- I've bound some durable object instances to some user id's, but those users no longer exist. The durable object used to store some user data. How can I remove all associated data within the durable object, without removing the entirety of the object itself? There doesn't seem to be a data viewer like KV when i can manually remove the key/value pairs.

E

Eric•1/20/25, 1:56 AM

Reading the docs, it seems like the data wont delete after becoming stale. Any durable objects only remove on shutdown + once they no longer have data within them.

EEric Reading the docs, it seems like the data wont delete after becoming stale. Any d...

1

1984 Ford Laser•1/20/25, 2:33 AM

You will need to do a

state.storage.deleteAll()

state.storage.deleteAll()

state.storage.deleteAll()

state.storage.deleteAll() to remove an old instance. An old instance will not incur any request or duration charges if they aren't being accessed, but their used storage will be charged until deleted

森

森@VerseLink (RW)•1/20/25, 9:28 AM

oh huh i see ensureReplicas() in the DO storage expirmental API, is this some future planned item?

森森@VerseLink (RW)oh huh i see ensureReplicas() in the DO storage expirmental API, is this some fu...

L

lambrospetrou•1/20/25, 9:37 AM

First use-case is for D1, which hopefully will be out soon-ish.

森

森@VerseLink (RW)•1/20/25, 9:38 AM

, does this mean the DO eventually will also have this feature

DDarkstar I've been looking all over for an example of how to do a Hibernatable DO-based W...

1

1984 Ford Laser•1/20/25, 11:38 AM

I'm pretty sure there's a note in the docs that hibernation only works with ingoing WS requests. I'll try and find the reference in the morning

DDarkstar I adjusted my code to try this: ``` const response = await fetch(gatewayUrl, { ...

1

1984 Ford Laser•1/20/25, 10:31 PM

I was able to find this note:

Hibernation is only supported when a Durable Object acts as a WebSocket server. Currently, outgoing WebSockets cannot hibernate.

Hibernation is only supported when a Durable Object acts as a WebSocket server. Currently, outgoing WebSockets cannot hibernate.

Hibernation is only supported when a Durable Object acts as a WebSocket server. Currently, outgoing WebSockets cannot hibernate.

Hibernation is only supported when a Durable Object acts as a WebSocket server. Currently, outgoing WebSockets cannot hibernate.

on this page here: https://developers.cloudflare.com/durable-objects/best-practices/websockets/#websocket-hibernation-api

1

1984 Ford Laser•1/20/25, 10:47 PM

When you do a

storage.get()

storage.get()

storage.get()

storage.get() and provide multiple keys, does each key count as a read unit, or do we count units per 4KB for all keys combined? Trying to minimise storage usage without moving to SQL backend

It'd be nice if SQLite DO can have a sync version of the KV API, even if it does exactly the same th

Similar Threads