I'm actually writing similar migration tool myself (for internally stuff), would you say that creati

森森@VerseLink (RW)I'm actually writing similar migration tool myself (for internally stuff), would...

L

There isn't better or worse. They are the same underlying storage. KV API goes to a special table that your SQL statements cannot modify, hence why I chose that. Otherwise you could mess up one of your actual migrations and delete your schema migration tracking table. Other than that separation, nothing else is different.

森

森@VerseLink (RW)OP•1/19/25, 9:07 AM

ah i see, thanks for the insight

森

森@VerseLink (RW)OP•1/19/25, 9:07 AM

I'm planning to keep track of the bookmarks when the migration happens (in case something bad happens, a rollback can be triggered immediately), so I would need an internal table to track that anyways

森森@VerseLink (RW)I'm planning to keep track of the bookmarks when the migration happens (in case ...

L

lambrospetrou•1/19/25, 9:09 AM

Choose the one that your code will be easier to read with and be correct

Both work.

Llambrospetrou Choose the one that your code will be easier to read with and be correct 🙂 Both...

森

森@VerseLink (RW)OP•1/19/25, 9:12 AM

oh also I'm also interested in using the export code that starbasedb offers, if the KV API Sql table is internal, does this mean we also need to export the KV data using list like original DO does?

森森@VerseLink (RW)oh also I'm also interested in using the export code that starbasedb offers, if ...

L

lambrospetrou•1/19/25, 9:13 AM

I haven't read all of the starbaseDB code, but if they only export SQL accessible data, I would assume that KV data is not included. Need to be tested though to be sure.

森

森@VerseLink (RW)OP•1/19/25, 9:14 AM

tldr it queries against sqlite_master

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

to get every table

森

森@VerseLink (RW)OP•1/19/25, 9:14 AM

not sure if the KV API's internal table is also tracked and can be queried against like this

森森@VerseLink (RW)tldr it queries against sqlite_master ```sql SELECT name FROM sqlite_master WHER...

L

lambrospetrou•1/19/25, 9:24 AM

You cannot access the KV table with SQL, just confirmed. There is also a test for it: https://github.com/cloudflare/workerd/blob/dbf8538fd806814acdfcbfec1f27c49b6f2f67b8/src/workerd/api/sql-test.js#L345

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

P

peachneo•1/19/25, 9:40 AM

do cloudflare devs have some internal library for testing DOs

Ppeachneo do cloudflare devs have some internal library for testing DOs

L

lambrospetrou•1/19/25, 9:42 AM

What do you mean? We use the public vitest integration too.

Llambrospetrou You cannot access the KV table with SQL, just confirmed. There is also a test fo...

森

森@VerseLink (RW)OP•1/19/25, 1:46 PM

:MeowHeartCloudflare:

Llambrospetrou because the KV API is the same for both types of DO, and in the non-SQLite one y...

B

Burrito•1/19/25, 4:26 PM

It'd be nice if SQLite DO can have a sync version of the KV API, even if it does exactly the same thing as the async one. I can get rid of the

blockConcurrencyWhile

blockConcurrencyWhile

when applying migration in constructor, and also I can put the entire migration into a

transactionSync

transactionSync

to prevent the rare event that migrations succeeded but writing migration number fails.

BBurrito It'd be nice if SQLite DO can have a sync version of the KV API, even if it does...

L

lambrospetrou•1/19/25, 4:35 PM

What would be the difference to use that instead of

transaction()

transaction()

?

blockConcurrencyWhile

blockConcurrencyWhile

has a different purpose than sync/async.

Llambrospetrou What would be the difference to use that instead of `transaction()`? `blockConcu...

B

Burrito•1/19/25, 4:51 PM

Say you have something like:

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

async function applyMigrations(storage: DurableObjectStorage) {
    await storage.get(...)

    // Do migration stuffs

    await storage.set(...)
}

In DO constructor:

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

class MyDO extends DurableObject {
    someState!: number

    constructor(ctx: DurableObjectState, env: Env) {
        super(ctx, env)

        console.log('before')
        applyMigrations(ctx.storage).then(() => {
            this.someState = 42
            console.log('initialized')
        })
        console.log('after')
    }
}

There are a few issues with this code:

Even if
```
storage.get
```
```
storage.get
```
/
```
storage.set
```
```
storage.set
```
's implementations are sync, because
```
applyMigration
```
```
applyMigration
```
is async,
```
.then
```
```
.then
```
puts continuation in the next event loop, so the log order will be
```
before
```
```
before
```
->
```
after
```
```
after
```
->
```
initialized
```
```
initialized
```
. I haven't tested it but unless DO has a different behavior to regular JS, that should be the case.
Because continuation goes on the next event loop, the DO is not fully initialized (
```
this.someState
```
```
this.someState
```
is still
```
undefined
```
```
undefined
```
) by the end of current event loop. I don't know if DO is able to process messages in the same event loop as the one doing the construction, but if that's true, then that's an issue.
Because initialization is in continuation, if you are using TS then you must null assert `someState!` because as far as TS can see, `this.someState` is not initialized when constructor exits.To fix some of these issues, you have to instead do:```tsclass MyDO extends DurableObject { someState!: number constructor(ctx: DurableObjectState, env: Env) { super(ctx, env) ctx.blockConcurrencyWhile(async () => { await applyMigrations(ctx.storage) this.someState = 42 }) }}```(This however still doesn't fix the last issue and you still have to `someState!`)If a sync KV API is exposed, this entire problem no longer exists.

森

森@VerseLink (RW)OP•1/19/25, 4:54 PM

I mean because KV API is actually sync, the blockConcurrencyWhile should complete execution kind of sync like, so it really isn't a issue

森

森@VerseLink (RW)OP•1/19/25, 4:54 PM

though the code looks a bit more uglier

B

Burrito•1/19/25, 4:56 PM

Yeah

blockConcurrencyWhile

blockConcurrencyWhile

is a workaround, but if we had an actual sync API, we wouldn't even need it.

BBurrito Say you have something like: ```ts async function applyMigrations(storage: Durab...

L

lambrospetrou•1/19/25, 4:56 PM

Your usecase seems to only need that due to Typescript typing. You can set a dummy value

this.someState = -1

this.someState = -1

before the call to

ctx.blockConcurrencyWhile(...)

ctx.blockConcurrencyWhile(...)

and then the property won't be considered optional.

Llambrospetrou Your usecase seems to only need that due to Typescript typing. You can set a dum...

B

Burrito•1/19/25, 4:58 PM

Sure, getting rid of that TS error is simple, but the point is that if we had a sync API we wouldn't need

blockConcurrencyWhile

blockConcurrencyWhile

at all, because

applyMigrations

applyMigrations

can just be sync.

B

Burrito•1/19/25, 5:01 PM

And because

storage.get

storage.get

/

storage.set

storage.set

is async, you also couldn't do:

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

async function applyMigrations(storage: DurableObjectStorage) {
    storage.transactionSync(async () => {
        const currentMigration = await storage.get(...)

        // Do migration stuffs

        await storage.set(...)
    })
}

Because just like the same problem above,

await

await

puts continuation in the next event loop, by which point

transactionSync

transactionSync

already exited.

BBurrito And because `storage.get`/`storage.set` is async, you also couldn't do: ```ts as...

L

lambrospetrou•1/19/25, 5:02 PM

That's why you also have

transaction(...)

transaction(...)

which is async. Anyway, we seem to be going in circles

B

Burrito•1/19/25, 5:02 PM

Can you use

transaction

transaction

with SQLite?

BBurrito Can you use `transaction` with SQLite?

L

lambrospetrou•1/19/25, 5:03 PM

Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction

When using the SQLite-backed storage engine, the txn object is obsolete. Any storage operations performed directly on the ctx.storage object, including SQL queries using ctx.storage.sql.exec(), will be considered part of the transaction.

You can use both APIs, KV and SQL, inside an SQLite DO.

B

Burrito•1/19/25, 5:04 PM

Ah, for some reason I was under the impression that I must use

transactionSync

transactionSync

with SQLite.

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

森

森@VerseLink (RW)OP•1/19/25, 5:04 PM

btw does trx.rollback() work with SQL storage backend?

B

Burrito•1/19/25, 5:05 PM

Cool, that gets rid of one part of the pain points then.

森

森@VerseLink (RW)OP•1/19/25, 5:05 PM

or do we just throw exceptions

Llambrospetrou Of course! https://developers.cloudflare.com/durable-objects/api/storage-api/#tr...

H

Hello, I’m Allie!•1/19/25, 6:23 PM

Can you do transactions across different I/O events?

HHello, I’m Allie!Can you do transactions across different I/O events?

L

lambrospetrou•1/19/25, 6:40 PM

What do you refer to as IO event?

森

森@VerseLink (RW)OP•1/19/25, 6:41 PM

btw it says stream api is only available in request context, does DOs RPC method consider a request context?

森

森@VerseLink (RW)OP•1/19/25, 7:46 PM

huh does DO have in-memory size limit?

Llambrospetrou What do you refer to as IO event?

H

Hello, I’m Allie!•1/19/25, 8:01 PM

Like, can I start a transaction within a fetch handler, then close it out within an RPC?

H

Hello, I’m Allie!•1/19/25, 8:01 PM

Or another combo of RPC/fetch/WebSocketMessage, etc

森森@VerseLink (RW)huh does DO have in-memory size limit?

H

Hello, I’m Allie!•1/19/25, 8:08 PM

Yes, same as a Worker, 128 MB

HHello, I’m Allie!Yes, same as a Worker, 128 MB

森

森@VerseLink (RW)OP•1/19/25, 8:09 PM

oh i thought they meant DO will always occupy 128MB, didn't know that is also the limit

S

Skrrrrrrrrr•1/19/25, 8:48 PM

is there some issues with the Turkey server?

HHello, I’m Allie!Like, can I start a transaction within a fetch handler, then close it out within...

L

lambrospetrou•1/19/25, 8:49 PM

The way you do an async transaction now is through

transaction(...)

transaction(...)

, so what you suggest could be done by starting one in one fetch/rpc call, but without awaiting the returned promise and just storing it in-memory. Then another request can await for it when needed.
This should work, but again, consistency is up to you to make sure what you do inside the transaction call is OK.
For example, if you try to iterate a cursor across IO context boundaries, I think it won't work (well-defined).

Llambrospetrou The way you do an async transaction now is through `transaction(...)`, so what y...

H

Hello, I’m Allie!•1/19/25, 8:51 PM

But you wouldn't be able to close out the transaction with context from the second event, right?

HHello, I’m Allie!But you wouldn't be able to close out the transaction with context from the seco...

L

lambrospetrou•1/19/25, 8:52 PM

What do you mean "close" the transaction? The transaction is a callback that you pass to the

transaction()

transaction()

method.

Llambrospetrou What do you mean "close" the transaction? The transaction is a callback that you...

森

森@VerseLink (RW)OP•1/19/25, 8:59 PM

I guess they meant creating a promise in the transaction and store the resolve function elsewhere (so the transaction keeps waiting) until it is the resolve function is resolved on a different / second event.
(But wouldn't that be a deadlock?)

森

森@VerseLink (RW)OP•1/19/25, 9:02 PM

@lambrospetrou hey, quick question, is it possible to create a Mock DO using vitest to test for functionality that relies on DO's SQL storage?
Mock DO as in, a dummy DO that only exists to provide it's SQL storage logic

森森@VerseLink (RW)@lambrospetrou hey, quick question, is it possible to create a Mock DO using vit...

L

lambrospetrou•1/19/25, 9:04 PM

https://github.com/cloudflare/workers-sdk/blob/main/fixtures/vitest-pool-workers-examples/durable-objects/test/direct-access.test.ts

Llambrospetrou What do you mean "close" the transaction? The transaction is a callback that you...

H

Hello, I’m Allie!•1/19/25, 9:06 PM

For example, if within the first fetch/rpc I do

// Using KV for simplicity
const a = txn.get("foo")

// Using KV for simplicity
const a = txn.get("foo")

// Using KV for simplicity
const a = txn.get("foo")

// Using KV for simplicity
const a = txn.get("foo")

I wouldn't be able to do this in the second fetch/rpc

txn.put("bar", a + "yo");

txn.put("bar", a + "yo");

txn.put("bar", a + "yo");

txn.put("bar", a + "yo");

H

Hello, I’m Allie!•1/19/25, 9:07 PM

Actually, this example doesn't really display a need for a transaction, but I meant more that you can't start a transaction within one I/O context, then complete it within another with data received in that context

HHello, I’m Allie!For example, if within the first fetch/rpc I do ```ts // Using KV for simplicity...

L

lambrospetrou•1/19/25, 9:12 PM

I still don't see what usecase would be good for blocking your transactions across requests. You just increase the likelyhood of having them fail due to conflicts at best. If I remember correctly, DOs have their own context, so in theory you can do what you say.

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

function doX() {
  this.promiseTxn = this.storage.transaction(async (txn) => {
    const a = txn.get("keyA");
    const nextRequestValue = await this.waitForAValueSetByAFutureRequest();
    txn.put("keyB", a + nextRequestValue);
  });
}

I "think" this should be doable, but I didn't confirm/test it. I would never do it though

Llambrospetrou https://github.com/cloudflare/workers-sdk/blob/main/fixtures/vitest-pool-workers...

森

森@VerseLink (RW)OP•1/19/25, 9:16 PM

hmm, my question is more like is it possible to have a dummy / mock DO class in the test folder, instead of in src folder that interferes with the exports of my actual app.

If not, would you suggest instead to create a different isolated package (for example in a monorepo)?

森森@VerseLink (RW)hmm, my question is more like is it possible to have a dummy / mock DO class in ...

L

lambrospetrou•1/19/25, 9:17 PM

I don't understand what you are asking. You can create your own worker and its Durable Objects bind to it with miniflare if you want and use that.

Llambrospetrou I don't understand what you are asking. You can create your own worker and its D...

森

森@VerseLink (RW)OP•1/19/25, 9:30 PM

Wait, actually thank you! Your hint of "miniflare" and that github example link really helped out big time!

森

森@VerseLink (RW)OP•1/19/25, 10:05 PM

hmm now i'm getting

Error: Durable Object test helpers can only be used with stubs pointing to objects defined within the same worker.

I'm actually writing similar migration tool myself (for internally stuff), would you say that creati

Similar Threads