If you see my SQLSchemaMigrations helper, I don't need to read any rows, except 1 integer at constru

If you see my SQLSchemaMigrations helper, I don't need to read any rows, except 1 integer at constructor time. It tracks up to which migration last succeeded, so next time, if no new migrations exist it doesn't do anything.

See https://github.com/lambrospetrou/durable-utils#readme

James•1/17/25, 4:23 PM

ah I see, there's an assumption that all migrations succeeded up to X, which is likely fine most of the time

JJames ah I see, there's an assumption that all migrations succeeded up to X, which is ...

lambrospetrouOP•1/17/25, 4:24 PM

what do you mean? It's not an assumption, they did succeed up to X otherwise the counter stored wouldn't be X.
if you go and modify specific migrations after they already ran once, that's on you.

James•1/17/25, 4:27 PM

right, its an edge case

in my case, i prefer to explicitly check for all migrations by name, but I don't check for or run migrations on every init or request, I have a failsafe which runs migratoins on schema errors, as well as a migration script to propagate migrations globally on demand or trigger then on next run. I figure migrations themselves are not something I want my DOs processing most of the time

JJames right, its an edge case in my case, i prefer to explicitly check for all migrat...

lambrospetrouOP•1/17/25, 4:27 PM

I don't think this is any different (name vs id), but anyway. As long as you are happy, we are happy

James•1/17/25, 4:43 PM

small thing, but why would you await the migration storage calls? ideally you want all DO SQL calls in the same request to run synch

JJames small thing, but why would you await the migration storage calls? ideally you wa...

lambrospetrouOP•1/17/25, 4:47 PM

Easier to reason about. If you add an async IO by mistake in-between your non-awaited storage operations then you might have consistency issues. If everything in your code is sync, yes you can avoid it. I prefer safe and correct code first.

Llambrospetrou I meant to deploy without that class being a binding on any worker, so that it's...

oscar•1/18/25, 5:53 PM

Not sure I’m following here. My initial bug happened with the DO class already having no bindings to any worker. The deleted_classes migration was rejected with the DO binding already removed.

JJacob Wright Oh, no, it is just that anyone can sign up for my service (just like anyone can ...

Moh•1/18/25, 11:12 PM

Not sure if plausible, but maybe you could store document IDs in a normal D1 db or KV, and validate before setting up your DO SQLite db?

Moh•1/18/25, 11:13 PM

Make creating a document an explicit command in your application, and you get to rate limit it, cap it etc, e.g. give users 100 documents limit, force them to delete the empty documents they used before

peachneo•1/19/25, 5:11 AM

has anyone else ran into "Durable Object storage is no longer accessible" in tests?

Ppeachneo has anyone else ran into "Durable Object storage is no longer accessible" in tes...

1984 Ford Laser•1/19/25, 5:12 AM

Are you trying to retrieve a key/value that was removed? Bit more context would be helpful

Llambrospetrou If you see my SQLSchemaMigrations helper, I don't need to read any rows, except ...

森

森@VerseLink (RW)•1/19/25, 5:13 AM

reading through the source code

森

森@VerseLink (RW)•1/19/25, 5:13 AM

await this.#_config.doStorage.put<number>(this.#_lastMigrationIDKeyName(), this.#_lastMigrationMonotonicId);

await this.#_config.doStorage.put<number>(this.#_lastMigrationIDKeyName(), this.#_lastMigrationMonotonicId);

森

森@VerseLink (RW)•1/19/25, 5:13 AM

shouldn't using transactionSync be enough?

森

森@VerseLink (RW)•1/19/25, 5:14 AM

since put / get is actually using special Sqlite table, and sqlite in DO is sync based, meaning there's no reason to ever await

森

森@VerseLink (RW)•1/19/25, 5:15 AM

https://developers.cloudflare.com/durable-objects/api/sql-storage/

Specifically for Durable Object classes with SQLite storage backend, KV operations which were previously asynchronous (for example, get, put, delete, deleteAll, list) are synchronous, even though they return promises. These methods will have completed their operations before they return the promise.

11984 Ford Laser Are you trying to retrieve a key/value that was removed? Bit more context would ...

peachneo•1/19/25, 5:18 AM

No... I'm running into a strange issue.

This works:

let ran = false;
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = false

let ran = false;
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = true
ran = await runDurableObjectAlarm(stub) // ran = false

And this - which as far as I can tell is functionally identical.. does not work (throws the given error)

while (true) {
    let ran = await runDurableObjectAlarm(stub)
    if (!ran) {
      break
    };
}

while (true) {
    let ran = await runDurableObjectAlarm(stub)
    if (!ran) {
      break
    };
}

Ppeachneo No... I'm running into a strange issue. This works: ``` let ran = false; ran = ...

1984 Ford Laser•1/19/25, 5:19 AM

does

runDurableObjectAlarm()

runDurableObjectAlarm()

return

true

true

or a similar value?

11984 Ford Laser does `runDurableObjectAlarm()` return `true` or a similar value?

peachneo•1/19/25, 5:19 AM

it returns a bool indicating whether the alarm executed or not

1984 Ford Laser•1/19/25, 5:21 AM

instead of

while (true)

while (true)

what about

while (ran)

while (ran)

? Declare

ran

ran

as true before the first run

11984 Ford Laser instead of `while (true)` what about `while (ran)`? Declare `ran` as true before...

peachneo•1/19/25, 5:23 AM

same result

1984 Ford Laser•1/19/25, 5:23 AM

Check what the alarm is returning to make sure its a bool?

11984 Ford Laser Check what the alarm is returning to make sure its a bool?

peachneo•1/19/25, 5:24 AM

yeah I've checked it's definitely a bool

1984 Ford Laser•1/19/25, 5:25 AM

What does the alarm do? Is the error in the alarm handler itself or this other code?

peachneo•1/19/25, 5:26 AM

it fails on a call to storage that the callback for the alarm calls

1984 Ford Laser•1/19/25, 5:29 AM

So then I assume the storage you're requesting isn't there? KV storage?

peachneo•1/19/25, 5:30 AM

durable object storage. Yeah I guess the storage isn't there... but the question is why lol

I mean how on earth does using a while loop that makes the exact same calls as the inlined non-loop version does result in a different outcome

peachneo•1/19/25, 5:31 AM

Smells like some weird race condition to me

Ppeachneo durable object storage. Yeah I guess the storage isn't there... but the question...

1984 Ford Laser•1/19/25, 5:33 AM

Very odd

peachneo•1/19/25, 6:26 AM

seems to be a timing issue, if I stick an arbitrary delay in the loop it passes

Niveth•1/19/25, 7:55 AM

What is the maximum time, I can set an alarm in durable object ?

NNiveth What is the maximum time, I can set an alarm in durable object ?

1984 Ford Laser•1/19/25, 9:00 AM

Maybe 03:14:07 UTC on 19 January 2038?

森森@VerseLink (RW)shouldn't using transactionSync be enough?

lambrospetrouOP•1/19/25, 9:00 AM

The KV API is async though, even though in SQlite DOs all storage operations are the same, so I am using

await

await

to be consistent with the API that you need to await a Promise. The underlying behavior doesn't change.
Someone reading the code and not seeing an

await

await

of a promise will be more confused, rather than having to read all the DO docs to figure out that it doesn't matter.

Llambrospetrou The KV API is async though, even though in SQlite DOs all storage operations are...

森

森@VerseLink (RW)•1/19/25, 9:03 AM

sounds good, yeah, I actually did notice that, I'm just thinking if the choice was intentional and if I have mistaken it or not

森

森@VerseLink (RW)•1/19/25, 9:03 AM

though, why doesn't DO expose a sync version of it under sql

森

森@VerseLink (RW)•1/19/25, 9:04 AM

i understand it's backwards compatiblity, but old DO can't migrate to sql backed DO though?

森森@VerseLink (RW)though, why doesn't DO expose a sync version of it under sql

lambrospetrouOP•1/19/25, 9:04 AM

because the KV API is the same for both types of DO, and in the non-SQLite one you actually have to await.

森

森@VerseLink (RW)•1/19/25, 9:05 AM

I'm actually writing similar migration tool myself (for internally stuff), would you say that creating a version table myself and use the sync api be better then KV API

森森@VerseLink (RW)I'm actually writing similar migration tool myself (for internally stuff), would...

lambrospetrouOP•1/19/25, 9:06 AM

There isn't better or worse. They are the same underlying storage. KV API goes to a special table that your SQL statements cannot modify, hence why I chose that. Otherwise you could mess up one of your actual migrations and delete your schema migration tracking table. Other than that separation, nothing else is different.

森

森@VerseLink (RW)•1/19/25, 9:07 AM

ah i see, thanks for the insight

森

森@VerseLink (RW)•1/19/25, 9:07 AM

I'm planning to keep track of the bookmarks when the migration happens (in case something bad happens, a rollback can be triggered immediately), so I would need an internal table to track that anyways

森森@VerseLink (RW)I'm planning to keep track of the bookmarks when the migration happens (in case ...

lambrospetrouOP•1/19/25, 9:09 AM

Choose the one that your code will be easier to read with and be correct

Both work.

Llambrospetrou Choose the one that your code will be easier to read with and be correct 🙂 Both...

森

森@VerseLink (RW)•1/19/25, 9:12 AM

oh also I'm also interested in using the export code that starbasedb offers, if the KV API Sql table is internal, does this mean we also need to export the KV data using list like original DO does?

森森@VerseLink (RW)oh also I'm also interested in using the export code that starbasedb offers, if ...

lambrospetrouOP•1/19/25, 9:13 AM

I haven't read all of the starbaseDB code, but if they only export SQL accessible data, I would assume that KV data is not included. Need to be tested though to be sure.

森

森@VerseLink (RW)•1/19/25, 9:14 AM

tldr it queries against sqlite_master

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

SELECT name FROM sqlite_master WHERE type='table' AND name=?;

to get every table

森

森@VerseLink (RW)•1/19/25, 9:14 AM

not sure if the KV API's internal table is also tracked and can be queried against like this

森森@VerseLink (RW)tldr it queries against sqlite_master ```sql SELECT name FROM sqlite_master WHER...

lambrospetrouOP•1/19/25, 9:24 AM

You cannot access the KV table with SQL, just confirmed. There is also a test for it: https://github.com/cloudflare/workerd/blob/dbf8538fd806814acdfcbfec1f27c49b6f2f67b8/src/workerd/api/sql-test.js#L345

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

> select * from _cf_KV;
access to _cf_KV.key is prohibited: SQLITE_AUTH

peachneo•1/19/25, 9:40 AM

do cloudflare devs have some internal library for testing DOs

Ppeachneo do cloudflare devs have some internal library for testing DOs

lambrospetrouOP•1/19/25, 9:42 AM

What do you mean? We use the public vitest integration too.

Llambrospetrou You cannot access the KV table with SQL, just confirmed. There is also a test fo...

森

森@VerseLink (RW)•1/19/25, 1:46 PM

:MeowHeartCloudflare:

If you see my SQLSchemaMigrations helper, I don't need to read any rows, except 1 integer at constru

Similar Threads

Similar Threads

Similar Threads