So, I want to know because durable object has 1GB limit per object. Let's say if I store a user data

So, I want to know because durable object has 1GB limit per object. Let's say if I store a user data per object in durable object, then later if I want to fetch let's say 20 user data all at once (for listing on frontend). How to do that

AAshish Rawat So, I want to know because durable object has 1GB limit per object. Let's say if...

James•10/5/24, 5:51 AM

you need a deterministic way of accessing your durable objects, likely a central db keeping track of IDs, you'd have to hit every stub with whatever query you want and join the results

Ashish RawatOP•10/5/24, 5:52 AM

so basically aggregate and manual sharding via timestamp

Ashish RawatOP•10/5/24, 5:53 AM

then I'll wait for durable object with sqlite to get more mature until sharding is automatic. The same is the case with D1.

Ashish RawatOP•10/5/24, 5:54 AM

This is very confusing for sqlite

Screenshot_2024-10-05_at_11.24.17_AM.png

matt•10/5/24, 6:39 AM

can you test a durable object locally without running workerd? perhaps just a plain JS implementation of the parent DurableObject class?

Mmatt can you test a durable object locally without running workerd? perhaps just a pl...

Hello, I’m Allie!•10/5/24, 6:49 AM

You can run Miniflare/wrangler yourself, but it still uses workerd under the hood

Hello, I’m Allie!•10/5/24, 6:50 AM

May I ask why specifically you want to avoid workerd?

matt•10/5/24, 6:54 AM

im migrating to bun and normally tests run so fast you can enable save on keypress in vscode and watch your tests run as you're typing

matt•10/5/24, 6:55 AM

but you wouldn't want any external libs (workerd) spinning up on each keypress. i think the way i'll get around it is taking all the business logic out of the DO into a regular class and just test that

matt•10/5/24, 6:56 AM

https://bun.sh/docs/runtime/hot

Mmatt but you wouldn't want any external libs (workerd) spinning up on each keypress. ...

Hello, I’m Allie!•10/5/24, 7:08 AM

Depending on what parts of a DO you use, you may be able to get it to work?

Hello, I’m Allie!•10/5/24, 7:08 AM

Though something like Synchronous SQLite probably isn’t stubable?

matt•10/5/24, 7:12 AM

i do use alarms but I need to stub those anyway so im splitting the BL into a seperate class and then will create a DurableObjectStub that wraps the BL class for testing, and the traditional DO class will also pull in the same BL class

matt•10/5/24, 7:12 AM

bun has an embedded sqlite too, so i'll use that in the test stub

Mmatt bun has an embedded sqlite too, so i'll use that in the test stub

Hello, I’m Allie!•10/5/24, 7:14 AM

Is it synchronous though? Since that does provide a bit more significant challenge

Hello, I’m Allie!•10/5/24, 7:15 AM

The other thing too that Bun doesn’t/can’t mock is the supported API surface

Hello, I’m Allie!•10/5/24, 7:15 AM

So some APIs available in

workerd

workerd

may be subtly different/present/not present in bunbun and vice versa

Hello, I’m Allie!•10/5/24, 7:16 AM

Your code may be able to compensate for that, but you would also need to make sure any dependencies you pull in also do so

matt•10/5/24, 7:18 AM

yes its synchronous - https://bun.sh/docs/api/sqlite

matt•10/5/24, 7:18 AM

i just migrated the first DO without any issue, but the other ones make use of more DO APIs so will have to just tackle them one by one and see if my stub can faithfully recreate a prod DO

matt•10/5/24, 7:19 AM

incredible feeling getting immediate feedback on your tests

Hoshenk13•10/5/24, 9:38 AM

lets say i want to connect more than 100k websockets with Hibernation.
since the limit is 32,768 WebSocket connections per Durable Object instance, i need some way monitor the count in every Durable Object instance, and create new objects and route requests to them? is it possible this way?

Original message was deleted

Hoshenk13•10/5/24, 10:07 AM

thanks for the recommendation.
i also need to broadcast events to all sockets on all DO's.

how can i do it?
maybe a route on each DO's, that will do broadcast and then i will request all DO's with thier ids?

Larry•10/5/24, 12:48 PM

@Matt , did you look into using the vitest DO integration for your testing?

LordSilver•10/5/24, 5:41 PM

DO is just a special Workers, so still runs in V8 Isolate?

LLordSilver DO is just a special Workers, so still runs in V8 Isolate?

Isaac McFadyen•10/5/24, 6:23 PM

Yes, that's correct.

IIsaac McFadyen Yes, that's correct.

LordSilver•10/5/24, 6:28 PM

So when you connect a Worker to a DO (another Worker), if they were in the same region, would that also be 0 latency?

LLordSilver So when you connect a Worker to a DO (another Worker), if they were in the same ...

Isaac McFadyen•10/5/24, 6:29 PM

Not necessarily. If a DO is already running on a server (metal) then the request would hit that metal, and there are many metals in a region so it might need to be sent to a different machine.

LordSilver•10/5/24, 6:29 PM

How is it different:
Worker => DO + SQLite
Worker => Workers KV

Workers KV is a normal Worker, is it a DO? Or anothing thing?

LLordSilver How is it different: Worker => DO + SQLite Worker => Workers KV Workers KV is a...

Hello, I’m Allie!•10/5/24, 6:30 PM

KV is a service running in the EU/US Core Datacenters + a datacenter-local cache

LordSilver•10/5/24, 6:30 PM

So it's not co-located with the Worker.

LordSilver•10/5/24, 6:30 PM

It's like a separate kv-store.

LordSilver•10/5/24, 6:30 PM

Kinda misleading that it's called Workers KV.

LordSilver•10/5/24, 6:31 PM

instead of Cloudflare KV

LLordSilver So it's not co-located with the Worker.

Hello, I’m Allie!•10/5/24, 6:31 PM

The data itself may be cached on the same metal, but that stored data is not treated as the source of truth, if that is what you mean

LLordSilver Kinda misleading that it's called Workers KV.

Hello, I’m Allie!•10/5/24, 6:31 PM

Not sure what you mean? It's Workers KV because it is KV for Workers, not necessarily KV built on Workers

Isaac McFadyen•10/5/24, 6:33 PM

^ in fact it is built on Workers now I believe, but that doesn't make it co-located with the Worker (even if the KV Worker was co-located the data is not - it's not realistic to replicate that much data to every metal/DC in the entire Cloudflare fleet)

LordSilver•10/5/24, 6:33 PM

Is SQLite inside DO similar to node:sqlite?

LordSilver•10/5/24, 6:33 PM

Basically the same thing but for workerd instead of node?

LLordSilver Is SQLite inside DO similar to [node:sqlite](https://nodejs.org/api/sqlite.html)...

Isaac McFadyen•10/5/24, 6:34 PM

Kind of? But each DO has it's own SQLite storage.

Hello, I’m Allie!•10/5/24, 6:34 PM

Some extra special sauce for Disaster-Recovery Replication / Point in Time Restores

Hello, I’m Allie!•10/5/24, 6:35 PM

At least AFAIK node:sqlitenode:sqlite doesn't do that out of the box

Hello, I’m Allie!•10/5/24, 6:35 PM

You can probably mod it to do it somehow...

LordSilver•10/5/24, 6:35 PM

It creates a sqlite db in memory.

LordSilver•10/5/24, 6:35 PM

Which I think is similar to what DO does since it says it lives in the same thread as the DO v8 isolate.

LLordSilver Click to see attachment

Isaac McFadyen•10/5/24, 6:36 PM

That's only that specific example.

Isaac McFadyen•10/5/24, 6:36 PM

It can operate on a disk based DB too.

LordSilver•10/5/24, 6:37 PM

only thing is that maybe DO would have a specific amount of resources reserved for the SQLite, whereas with node:sqlitenode:sqlite you would need to be careful in not exceeding your memory/disk.

LLordSilver only thing is that maybe DO would have a specific amount of resources reserved f...

Hello, I’m Allie!•10/5/24, 6:40 PM

IIRC DO SQLite execution shares the memory of your code, so that is still a concern there