Cloudflare Developers•16mo ago

Would it be possible to use drizzle / run drizzle migrations against the new DO sql database?

Kkenton (but in the DO case you have to first route the request to the DO from a worker,...

LordSilver•9/27/24, 1:25 PM

but the SQLite dbs living inside the DOs aren't replicating the same data, they are all separate dbs

LLordSilver but the SQLite dbs living inside the DOs aren't replicating the same data, they ...

kenton•9/27/24, 1:33 PM

Right, it works best if you can divide your data into many small databases. That said, we'll be introducing replication at some point.

Nnachotoday the blog post talks about how output gates will block a response being returned ...

kenton•9/27/24, 1:35 PM

Yes, it blocks all forms of output. By the way, output gates were previously introduced a few years back in this blog post: https://blog.cloudflare.com/durable-objects-easy-fast-correct-choose-three/

Agreed the docs ought to explain it better.

The Cloudflare Blog

Durable Objects: Easy, Fast, Correct — Choose three

When multiple clients access the same storage concurrently, race conditions abound. Durable Objects can make it easier. We recently rolled out improvements to Durable Objects that automatically correct many common race conditions while actually making your code faster.

Kkenton Right, it works best if you can divide your data into many small databases. That...

LordSilver•9/27/24, 1:51 PM

so that all DOs around the world have the internal SQLite DBs synced with eachother?

LordSilver•9/27/24, 1:52 PM

it would be similar to what Turso currently does on fly.io infra

LLordSilver so that all DOs around the world have the internal SQLite DBs synced with eachot...

kenton•9/27/24, 2:02 PM

When we implement replication, the idea will be that any particular DO can be replicated to multiple locations around the world. So if a lot of workers are requesting the same DO ID, they'll be routed to appropriate replicas, and the sqlite state is replicated from the primary to all those replicas. Durable Objects with different IDs won't sync with each other.

Kkenton When we implement replication, the idea will be that any particular DO can be re...

Hard@Work•9/27/24, 2:03 PM

Would the primaries be movable too?

HHard@Work Would the primaries be movable too?

kenton•9/27/24, 2:05 PM

Making DOs movable is a separate feature we want to work on. I think with the new storage backend it will be a lot easier for us to finally do that, but I think replication is likely to happen first.

Kkenton When we implement replication, the idea will be that any particular DO can be re...

LordSilver•9/27/24, 2:40 PM

D1 would become pointless at that point tho

Original message was deleted

kenton•9/27/24, 2:42 PM

already there: https://developers.cloudflare.com/durable-objects/api/storage-api/#point-in-time-recovery

Cloudflare Docs

Storage API | Cloudflare Durable Objects docs

The Storage API allows Durable Objects to access transactional and strongly consistent storage. A Durable Object’s attached storage is private to its unique instance and cannot be accessed by other objects.

LLordSilver D1 would become pointless at that point tho

kenton•9/27/24, 2:45 PM

It depends. Building for DO still means architecting in a new, different way that not everyone is up for. D1's goal is to support the classic three-tier web architecture that people are used to.

Kkenton It depends. Building for DO still means architecting in a new, different way tha...

LordSilver•9/27/24, 2:49 PM

but you are still using 3-tier in both cases

LordSilver•9/27/24, 2:49 PM

at least for the use case i described

LordSilver•9/27/24, 2:49 PM

not the one where you use sqlite just for storing user config like if it was a kv-store

Abakir•9/27/24, 3:48 PM

The Durable Object's code has been updated, this version can no longer access storage.

So i get this eror in my logs after pushing a new update, would having a retry mechanism in the fetch code fix this problem?

Original message was deleted

Abakir•9/27/24, 3:54 PM

Should I us a try catch, or making the computer connected to the websocket reconnect?

Kkenton The latency for D1 will depend on how far your worker is from the D1 database. I...

o7•9/27/24, 6:17 PM

Would D1 w/ Smart Placement’s latency be comparable to SQLite on DO’s latency?
I’m going to be doing a little test soon but I’m asking just to know what to expect with my results

Oo7 Would D1 w/ Smart Placement’s latency be comparable to SQLite on DO’s latency? I...

kenton•9/27/24, 6:20 PM

Smart Placement with D1 will reduce your latency down to a few milliseconds, but SQLite on DO has latency of a few microseconds. Still a big difference, but may or may not matter depending on how you use it.

Ashish Rawat•9/27/24, 6:40 PM

I want to ask how all DO will work together. For eg: Let's say I want to query across all DO on sqlite Because DO has 1 GB limit, I'm still not getting how to infra my app according to it, if I want to migrate from mongodb.

Ashish Rawat•9/27/24, 6:41 PM

There must be a fullstack tutorial for that with DO on sqlite3 with sharding

Original message was deleted

kenton•9/27/24, 7:25 PM

Yes, if you're on the sqlite storage backend, the KV API still works and is actually backed by a hidden table in SQLite, so you get all the benefits.

AAshish Rawat I want to ask how all DO will work together. For eg: Let's say I want to query a...

kenton•9/27/24, 7:27 PM

Generally you want to split your database into logical "shards" of related data, trying to make these shards as fine-grained as reasonably makes sense. Remember that it's totally fine to have millions of DOs -- the system is built for that.

kenton•9/27/24, 7:27 PM

one common strategy is to put each user's data in a separate DO. But the right split depends on the app.

shoo•9/27/24, 7:45 PM

Hey guys
I'm not sure if this is possible or if its a good idea

but because mongodb long connections are not possible for workers and the documentation suggests that we use mongodb realm.

is it possible to use durable objects to store the connection details and get that everytime a new worker is invoked?

Kkenton Yes, if you're on the sqlite storage backend, the KV API still works and is actu...

geg•9/27/24, 9:13 PM

Can I ask why is the table hidden?

Ggeg Can I ask why is the table hidden?

kenton•9/27/24, 9:17 PM

It contains V8-serialized values. We don't want people trying to decode the V8 serialization as it sometimes breaks in backwards-incompatible ways (which we can deal with as long as we control all the deserializers). We are also not completely sure that the parser is resilient against malicious input. Chrome similarly uses V8 serialization but never lets apps see nor modify the serialized bytes.

Kkenton It contains V8-serialized values. We don't want people trying to decode the V8 s...

geg•9/27/24, 9:23 PM

Thank you for the answer! I was particularly interested in the serialization/deserialization, that clears up why there's no functions for that exposed.

Wallacy•9/27/24, 11:48 PM

Still plans to put DOs on more regions? Sucks very hard here on South America. And i get the problem of the storage, but in many cases a DO whiout any storage backend will be just fine. For handle Websockets will be very helpfull, today i need to pay the latency to call a DO across the atlantic just make a websocket point. Sometimes i use a DO just to keep my in memory graphql cache, etc. The point is: Every point that runs a worker should be able to run a DO without storage. (more storage locations will be good also);

Vy•9/28/24, 12:21 AM

FYI we've identified an issue with alarms in the beta for SQLite in Durable Objects. While our team implements a fix, we are temporarily disabling alarms for only Durable Objects with SQLite storage backend.

Durable Objects with the default, key-value storage are unaffected and alarms are functioning normally

Pending docs:https://github.com/cloudflare/cloudflare-docs/pull/17185

GitHub

Disable alarms for SQLite in DO by vy-ton · Pull Request #17185 · c...

Disable alarms in SQLite in DO temporarily while a fix is implemented

Chaika•9/28/24, 12:29 AM

aw hopefully doesn't take too long for fix, was going to play around with alarms and sqlite

kchro3•9/28/24, 12:56 AM

hope it's ok to solicit feedback on a use-case here.

i'm interested in building a personalized email scheduler, and i want to see if DO would be a good architecture for this & if it would scale to millions of users. the exact use case is that i have a ranking algo where for a given user & set of emails, i'll predict which email to send, and now i want to predict the optimal send time too.

the way i've seen this work at prev companies has been to have a daily cron job that computes the optimal send time for every user & stores them in a database. then a separate cron job would run like hourly to fetch the users eligible to receive an email or notification and then apply the content ranking algorithm. however, that usually means that you'll need at least two cron jobs, a database, and a pubsub topic.

what i'd like to do is to use a durable object per user to represent their send time preferences. in an alarm routine, it would wake up => fetch & rank the user's eligible emails => send an email => set its next alarm. a separate worker would receive webhook events about when an email was sent / opened / clicked / etc, so it would update the durable object per user. if the email is opened, the worker could update the user's preferences & reschedule the alarm.

if this sounds like a bad idea (too overengineered, etc), would love to know before investing too much. i'd be looking at supporting millions of users & emails per week

Kkenton Generally you want to split your database into logical "shards" of related data,...

Ashish Rawat•9/28/24, 6:23 AM

Is there any way to create durable objects bindings on the fly with api or sdk for each users or maybe I'm missing something. A basic tutorial on scalibility would be appreciated if possible.

AAshish Rawat Is there any way to create durable objects bindings on the fly with api or sdk f...

Ashley•9/28/24, 9:15 AM

See https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-from-a-worker/, you would have a binding for a given DO (implemented by a class in your code) and then create DOs on the fly using whatever mechanism works for you (e.g. create a DO using a userId with idFromName). So the DO creation is dynamic, it’s not like other bindings where you create a resource and then you interact with one resource with an ID, with DOs you bind to a class + then generate millions of resources (if you need)

John Spurlock•9/29/24, 2:47 PM

important metrics for sqlite-backed DOs already in graphql, thanks!

JVictorV•9/29/24, 4:52 PM

Does anyone have a good way to uniquely identify websocket connections? I have a simple map of id -> websocket that i want to keep in sync even when the ws hibernates or the DO is evicted from memory

Matt Silverlock•9/29/24, 5:13 PM

Set tags when you accept the WebSockets: https://developers.cloudflare.com/durable-objects/api/websockets/#acceptwebsocket

Cloudflare Docs

WebSockets | Cloudflare Durable Objects docs

WebSockets are long-lived TCP connections that enable bi-directional, real-time communication between client and server.

JJikyu Would it be possible to use drizzle / run drizzle migrations against the new DO ...

zwily•9/29/24, 9:17 PM

same question here! drizzle may need to make another adapter for it

1.1.1.1APP•9/29/24, 10:01 PM

DNS over Discord: Invite

Invite DNS over Discord to your server with https://dns-over-discord.v4.wtf/invite

Or, use the bot anywhere by adding it to your account with https://dns-over-discord.v4.wtf/invite/user

Nick•9/30/24, 6:07 AM

Is there any way to get notified when a new DO is created or updated? I guess you can build it yourself with Alarms, but wondering if something built in