How is it different: Worker => DO + SQLite Worker => Workers KV Workers KV is a normal Worker, is i

How is it different:
Worker => DO + SQLite
Worker => Workers KV

Workers KV is a normal Worker, is it a DO? Or anothing thing?

LLordSilver How is it different: Worker => DO + SQLite Worker => Workers KV Workers KV is a...

Hello, I’m Allie!•10/5/24, 6:30 PM

KV is a service running in the EU/US Core Datacenters + a datacenter-local cache

LordSilverOP•10/5/24, 6:30 PM

So it's not co-located with the Worker.

LordSilverOP•10/5/24, 6:30 PM

It's like a separate kv-store.

LordSilverOP•10/5/24, 6:30 PM

Kinda misleading that it's called Workers KV.

LordSilverOP•10/5/24, 6:31 PM

instead of Cloudflare KV

LLordSilver So it's not co-located with the Worker.

Hello, I’m Allie!•10/5/24, 6:31 PM

The data itself may be cached on the same metal, but that stored data is not treated as the source of truth, if that is what you mean

LLordSilver Kinda misleading that it's called Workers KV.

Hello, I’m Allie!•10/5/24, 6:31 PM

Not sure what you mean? It's Workers KV because it is KV for Workers, not necessarily KV built on Workers

Isaac McFadyen•10/5/24, 6:33 PM

^ in fact it is built on Workers now I believe, but that doesn't make it co-located with the Worker (even if the KV Worker was co-located the data is not - it's not realistic to replicate that much data to every metal/DC in the entire Cloudflare fleet)

LordSilverOP•10/5/24, 6:33 PM

Is SQLite inside DO similar to node:sqlite?

LordSilverOP•10/5/24, 6:33 PM

Basically the same thing but for workerd instead of node?

LLordSilver Is SQLite inside DO similar to [node:sqlite](https://nodejs.org/api/sqlite.html)...

Isaac McFadyen•10/5/24, 6:34 PM

Kind of? But each DO has it's own SQLite storage.

Hello, I’m Allie!•10/5/24, 6:34 PM

Some extra special sauce for Disaster-Recovery Replication / Point in Time Restores

Hello, I’m Allie!•10/5/24, 6:35 PM

At least AFAIK node:sqlitenode:sqlite doesn't do that out of the box

Hello, I’m Allie!•10/5/24, 6:35 PM

You can probably mod it to do it somehow...

LordSilverOP•10/5/24, 6:35 PM

It creates a sqlite db in memory.

LordSilverOP•10/5/24, 6:35 PM

Which I think is similar to what DO does since it says it lives in the same thread as the DO v8 isolate.

LLordSilver Click to see attachment

Isaac McFadyen•10/5/24, 6:36 PM

That's only that specific example.

Isaac McFadyen•10/5/24, 6:36 PM

It can operate on a disk based DB too.

LordSilverOP•10/5/24, 6:37 PM

only thing is that maybe DO would have a specific amount of resources reserved for the SQLite, whereas with node:sqlitenode:sqlite you would need to be careful in not exceeding your memory/disk.

LLordSilver only thing is that maybe DO would have a specific amount of resources reserved f...

Hello, I’m Allie!•10/5/24, 6:40 PM

IIRC DO SQLite execution shares the memory of your code, so that is still a concern there

LordSilverOP•10/5/24, 6:41 PM

but Cloudflare doesn't have any setting for specifying resources of your Worker/DO

LordSilverOP•10/5/24, 6:41 PM

you don't even know how much you are using

LLordSilver but Cloudflare doesn't have any setting for specifying resources of your Worker/...

Isaac McFadyen•10/5/24, 6:41 PM

All Workers/DOs get up to 128mb which may be shared with other invocations of the same Worker

LordSilverOP•10/5/24, 6:41 PM

ah right

Isaac McFadyen•10/5/24, 6:41 PM

And DO SQLite has a size limit of 1GB during beta.

LordSilverOP•10/5/24, 6:42 PM

1GB of disk or memory?

LLordSilver 1GB of disk or memory?

Isaac McFadyen•10/5/24, 6:42 PM

Disk. Memory is shared with the DO invoking it.

LordSilverOP•10/5/24, 6:43 PM

So all DOs have 1GB of disk even if you aren't using the SQLite but want to save something else on it?

LLordSilver So all DOs have 1GB of disk even if you aren't using the SQLite but want to save...

Isaac McFadyen•10/5/24, 6:43 PM

No, SQLite storage is seperate. DOs don't have disk, they have SQLite (1GB) or DO KV storage.

Isaac McFadyen•10/5/24, 6:43 PM

In general you shouldn't think about it as "disk" but more like "I have a SQLite DB that can have a maximum size of 1GB"

Isaac McFadyen•10/5/24, 6:43 PM

All of the underlying hardware resources like disk/memory/cpu are managed by Cloudflare.

LordSilverOP•10/5/24, 6:44 PM

so it's like Workers and the Browser API which lives in a different machine

LLordSilver so it's like Workers and the Browser API which lives in a different machine

Isaac McFadyen•10/5/24, 6:44 PM

No, it lives on the same machine.

LordSilverOP•10/5/24, 6:44 PM

yeah, i meant on the fact that you cannot use the resources of the API for doing other stuff

Isaac McFadyen•10/5/24, 6:44 PM

Yeah.

LordSilverOP•10/5/24, 6:45 PM

well, i think you could potentially still use the sqlite as an object storage and dumping raw binary files in it

Isaac McFadyen•10/5/24, 6:46 PM

Yes you could. Keep in mind that there is no streaming of SQLite values so it would be held in memory.

Isaac McFadyen•10/5/24, 6:46 PM

(unlike alternatives such as R2)

LordSilverOP•10/5/24, 6:48 PM

so is this SQLite better than a Redis

Elliot Hesp•10/5/24, 8:34 PM

How long do we think it'll be until the alarms issue is with the sqlite is resolved?

Built something cool locally and then saw the warning in the docs

LLordSilver so is this SQLite better than a Redis

Larry•10/5/24, 10:52 PM

Redis is key-value oriented so in that sense it's more like Workers KV or the DO KV API. However, it has more functionality than a simple KV store including things like pub/sub and atomic operations like appending to a string or incrementing a value (although you can also do the latter atomicly with the DO KV API. Redis is often used as an in-memory distributed cache (think alternative to memcache) but it also has persistence.

The most obvios difference to SQLite is... well... SQL. SQLite is embedded not distributed (although read-replicas are implemented by some systems which is going to be the case for Cloudflare's D1) which limits scalability. Redis is distributed and scales up well.

The DO SQLite API though has another huge advantage beyond vanilla SQLite. With input gates, output gates, and write coalescing, you can essentially write stored procedures in JavaScript with atomicity, consistency, isolation, and durability (ACID). This means that during a single request where you never await, any combination of reads followed by any combination of writes are executed without worrying about another request changing anything from underneath you and where you can be sure that either all writes will succeed or all writes will fail. Caveat that I haven't confirmed that if you get an error back from an SQL write operation all the writes before that will be rolled back but I suspect that's the case and I intend to verify that experimentally shortly.

Note, these "stored procedures" run not only on the same machine but in the same memory space and thread making them very efficient and fast per request.

The tradeoff is that a single DO instance has a max throughput of about 1000 requests per second. The scaling model is to have many DOs so depending upon your application, there could be some coding complexity on your part and efficiency costs to coordinate between them. That said, many applications have natural boundaries across which little coordination is needed.

TTobias Tag me again when we have drizzle for DO Sqlite

sadsciencee•10/5/24, 11:54 PM

@tobias not official driver but i got it working sort of https://x.com/sadsciencee/status/1842714039719432404

sadsciencee (@sadsciencee) on X

successfully duct taped @DrizzleORM onto the new @CloudflareDev durable objects sqlite api

technically DO sqlite is synchronous, i used drizzle's http proxy which requires async. but im gonna let someone smarter than me figure that part out ¯\_(ツ)_/¯

Twitter

•

10/5/24, 11:50 PM

Ssadsciencee @tobias not official driver but i got it working sort of https://x.com/sadscienc...

Tobias•10/6/24, 9:05 AM

I tried the same thing but got random errors hahah

Tobias•10/6/24, 9:05 AM

Have you tried the query syntax?

TTobias Have you tried the query syntax?

sadsciencee•10/6/24, 11:00 AM

ya - tbh havent done any relational or joins yet. but basic findFirst/findMany works

working on streaming in bulk import files to battle test writes rn, ill have enough of a db to try some more complex queries after that

Kkenton yes, this applies to `sql.exec()`.

Larry•10/6/24, 12:03 PM

@kenton , does it also apply when the request exits with an uncaught runtime JavaScript error? Do the ones coalesced so far not get executed?

LLarry @kenton , does it also apply when the request exits with an uncaught runtime Jav...

kenton•10/6/24, 1:49 PM

No, not necessarily. If you want to roll back on exceptions, you should wrap your code in

this.ctx.storage.transactionSync(() => {...})

this.ctx.storage.transactionSync(() => {...})

Kkenton No, not necessarily. If you want to roll back on exceptions, you should wrap you...

Larry•10/6/24, 2:26 PM

Thanks. I don't see

.transactionSync()

.transactionSync()

in the docs

LordSilverOP•10/6/24, 3:33 PM

Does native replica support in SQLite change anything for SQLite-in-DO? https://sqlite.org/draft/rsync.html