I’ve been thinking, the total account level limit for DO is 50GB, but D1 has limit of 250 GB; it was

森森@VerseLink (RW)I’ve been thinking, the total account level limit for DO is 50GB, but D1 has lim...

J

James•1/23/25, 1:47 AM

You bring up some good points. I’d always assumed DO account level storage limits would scale up by a huge factor once the service is GA. Part of the reason I moved most of my storage to DO was to get around D1 limits for storage. 10Gb isn’t enough, and if I’m scaling horizontally, it made more sense to move to DO sql db tenants.

JJames You bring up some good points. I’d always assumed DO account level storage limit...

森

森@VerseLink (RW)OP•1/23/25, 2:09 AM

Agreed, I started to explore DO sql because of D1 limits (and also explored turso), but when I understood the potential of DO with SQL, I really want to work some new projects on it, my biggest concern now has been that DOs artificial limitations made sense with old DO, but with what it could do with SQL, I think the restrictions needs to be heavily revised

H

HowlOftheSun•1/23/25, 4:39 AM

Is that code in a shareable place? I'd be interested in seeing what you did

森森@VerseLink (RW)I’ve been thinking, the total account level limit for DO is 50GB, but D1 has lim...

L

lambrospetrou•1/23/25, 8:26 AM

The account storage limit for DOs is purely a soft limit. If you ask for a raise you will get a raise, assuming you are not doing something obviously phishy on the platform. I am guessing the team will raise it for everyone at some point in the future too, but for now you can ask for a raise of it once you are close to it.

Aalgads Pleasantly surprised this evening. I wanted to see if the outbox pattern could ...

L

lambrospetrou•1/23/25, 8:33 AM

Is there any documentation about what features in sqlite are available within the DO context? Is it safe to assume all the big ones like views, triggers, etc.?

Unfortunately, the docs are a bit spread now (we are fixing them gradually), but you can see D1's SQL docs and they are the same for DOs too.

https://developers.cloudflare.com/d1/sql-api/sql-statements/ (pragmas you can use)
https://developers.cloudflare.com/d1/sql-api/foreign-keys/ (foreign key relevant things)
https://developers.cloudflare.com/d1/sql-api/query-json/ JSON

As of today DOs use SQLite 3.47.

Aalgads Pleasantly surprised this evening. I wanted to see if the outbox pattern could ...

森

森@VerseLink (RW)OP•1/23/25, 9:38 AM

here's a another question: (maybe similarly applying to D1 too), how is trigger execution priced? is it priced the same rate as DO or likely no charge at all

森

森@VerseLink (RW)OP•1/23/25, 9:39 AM

(also ignore the question i just deleted, i brain farted)

森森@VerseLink (RW)here's a another question: (maybe similarly applying to D1 too), how is trigger ...

L

lambrospetrou•1/23/25, 10:08 AM

There isn't separate pricing. As long as your DO runs, you are charged as usual.

A

arbfay•1/23/25, 11:46 AM

Flash DX improvement idea:

instead of

const doid = env.MY_DO.idFromString(user_id)
const stub = env.MY_DO.get(doid)

const doid = env.MY_DO.idFromString(user_id)
const stub = env.MY_DO.get(doid)

const doid = env.MY_DO.idFromString(user_id)
const stub = env.MY_DO.get(doid)

const doid = env.MY_DO.idFromString(user_id)
const stub = env.MY_DO.get(doid)

let stubs be retrieved with just a string (to reduce boilerplate)

const stub = env.MY_DO.get(user_id) // ID
// or, for names
const stub = env.MY_DO.get(user_id, "name") // like KV's type

const stub = env.MY_DO.get(user_id) // ID
// or, for names
const stub = env.MY_DO.get(user_id, "name") // like KV's type

const stub = env.MY_DO.get(user_id) // ID
// or, for names
const stub = env.MY_DO.get(user_id, "name") // like KV's type

const stub = env.MY_DO.get(user_id) // ID
// or, for names
const stub = env.MY_DO.get(user_id, "name") // like KV's type

just one less line to work with DOs

Aarbfay Flash DX improvement idea: instead of ```js const doid = env.MY_DO.idFromStrin...

M

Max (@rozenmd)•1/23/25, 12:15 PM

When you get a network disconnect and want to recreate the stub, you don't want to recompute the ID though

E

elunesarrow•1/23/25, 12:19 PM

Hi, let's say I'm using DO only to rpc into its sqlite data, no websocket session, do I get charge for anything outside rpc (i.e. startup time & idle time post-rpc if any?)

MMax (@rozenmd)When you get a network disconnect and want to recreate the stub, you don't want ...

A

arbfay•1/23/25, 12:20 PM

is it an expensive operation? and why should that be the developer's concern and not the binding/platform's concern?

Aarbfay is it an expensive operation? and why should that be the developer's concern and...

L

lambrospetrou•1/23/25, 12:34 PM

In reality most real-world applications do not expose directly the raw DO IDs as their own application IDs, so you anyway will have to do some conversion/validation/parsing before passing it to the

get(...)

get(...)

method. So, it's 1 extra line among tens/hundreds.

I do usually end up creating a helper function though that wraps my applications logic and achieves what you suggest above. 1 line to create DO stubs, but based on my apps parsing logic.

K

KWAC•1/23/25, 1:53 PM

why it closes connection when i send message to server?

// Global list to store all active WebSocket connections
const clients = new Set();

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  const upgradeHeader = request.headers.get('Upgrade');
  if (!upgradeHeader || upgradeHeader !== 'websocket') {
    return new Response('Expected Upgrade: websocket', { status: 426 });
  }

  const webSocketPair = new WebSocketPair();
  const [client, server] = Object.values(webSocketPair);

  // Add the new connection to the global list
  clients.add(server);
  server.accept();

  // Handle incoming messages
  server.addEventListener('message', event => {
    console.log('Message received from client:', event.data);

    // Broadcast the message to all connected clients
    clients.forEach(client => {
      if (client !== server && client.readyState === WebSocket.OPEN) {
        client.send(`Broadcast: ${event.data}`);
      }
    });
  });

  // Handle connection close
  server.addEventListener('close', () => {
    console.log('Client disconnected');
    // Remove the connection from the global list
    clients.delete(server);
  });

  return new Response(null, {
    status: 101,
    webSocket: client,
  });
}

// Global list to store all active WebSocket connections
const clients = new Set();

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  const upgradeHeader = request.headers.get('Upgrade');
  if (!upgradeHeader || upgradeHeader !== 'websocket') {
    return new Response('Expected Upgrade: websocket', { status: 426 });
  }

  const webSocketPair = new WebSocketPair();
  const [client, server] = Object.values(webSocketPair);

  // Add the new connection to the global list
  clients.add(server);
  server.accept();

  // Handle incoming messages
  server.addEventListener('message', event => {
    console.log('Message received from client:', event.data);

    // Broadcast the message to all connected clients
    clients.forEach(client => {
      if (client !== server && client.readyState === WebSocket.OPEN) {
        client.send(`Broadcast: ${event.data}`);
      }
    });
  });

  // Handle connection close
  server.addEventListener('close', () => {
    console.log('Client disconnected');
    // Remove the connection from the global list
    clients.delete(server);
  });

  return new Response(null, {
    status: 101,
    webSocket: client,
  });
}

// Global list to store all active WebSocket connections
const clients = new Set();

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  const upgradeHeader = request.headers.get('Upgrade');
  if (!upgradeHeader || upgradeHeader !== 'websocket') {
    return new Response('Expected Upgrade: websocket', { status: 426 });
  }

  const webSocketPair = new WebSocketPair();
  const [client, server] = Object.values(webSocketPair);

  // Add the new connection to the global list
  clients.add(server);
  server.accept();

  // Handle incoming messages
  server.addEventListener('message', event => {
    console.log('Message received from client:', event.data);

    // Broadcast the message to all connected clients
    clients.forEach(client => {
      if (client !== server && client.readyState === WebSocket.OPEN) {
        client.send(`Broadcast: ${event.data}`);
      }
    });
  });

  // Handle connection close
  server.addEventListener('close', () => {
    console.log('Client disconnected');
    // Remove the connection from the global list
    clients.delete(server);
  });

  return new Response(null, {
    status: 101,
    webSocket: client,
  });
}

// Global list to store all active WebSocket connections
const clients = new Set();

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  const upgradeHeader = request.headers.get('Upgrade');
  if (!upgradeHeader || upgradeHeader !== 'websocket') {
    return new Response('Expected Upgrade: websocket', { status: 426 });
  }

  const webSocketPair = new WebSocketPair();
  const [client, server] = Object.values(webSocketPair);

  // Add the new connection to the global list
  clients.add(server);
  server.accept();

  // Handle incoming messages
  server.addEventListener('message', event => {
    console.log('Message received from client:', event.data);

    // Broadcast the message to all connected clients
    clients.forEach(client => {
      if (client !== server && client.readyState === WebSocket.OPEN) {
        client.send(`Broadcast: ${event.data}`);
      }
    });
  });

  // Handle connection close
  server.addEventListener('close', () => {
    console.log('Client disconnected');
    // Remove the connection from the global list
    clients.delete(server);
  });

  return new Response(null, {
    status: 101,
    webSocket: client,
  });
}

Aalgads Pleasantly surprised this evening. I wanted to see if the outbox pattern could ...

J

James•1/23/25, 3:02 PM

Id work to ensure heavy compute takes place away from the DO due to cost. You pay for wall time on a DO but not a worker, so api calls are much cheaper outside a DO. You could adjust your logic to still use triggers on the DO but offload to a worker once the alarm kicks off.

Aalgads Here is sample code that should give a good idea of what I am trying to do: http...

H

HowlOftheSun•1/23/25, 3:32 PM

Thanks!

J

James•1/23/25, 4:08 PM

@algads , caveat is that you lose the synchronicity of a single threaded DO once you offload to a worker, events may push to your API out of order

Aalgads @Ahz thinking about this some more though, since the DO is single threaded and t...

J

James•1/23/25, 5:00 PM

not if downstream workers are delayed or blocked, req1 to API hangs, req2 5 seconds later finishes first.

J

James•1/23/25, 5:14 PM

it could fire again, but the DO will be busy on its current thread, but if you await anywhere in the DO, then you're open to losing synch

and if you fire the alarm and offload to a worker, well thats all async and anything goes

JJames it could fire again, but the DO will be busy on its current thread, but if you a...

L

lambrospetrou•1/23/25, 5:53 PM

That's not how it would work. Each time you set the alarm you schedule one execution. And you setup a new alarm in that invocation, so until you set a new alarm time, there won't be a new alarm invocation.

J

James•1/23/25, 6:09 PM

are you sure we're talking about the same thing?

alarm 1 fires, has async tasks and takes 10 seconds with some async sprinkled throughout
during the execution context switching, other things can happen on the DO

I know only one alarm can be set at a time, but in this case the alarm already fired, and its up to the developer to manage the data flow from there, especially if these tasks run async

JJames are you sure we're talking about the same thing? alarm 1 fires, has async tasks...

L

lambrospetrou•1/23/25, 6:10 PM

For multiple async requests sure, it's the developer, but previous messages were arguing about having two alarm invocations at the same time.

Llambrospetrou For multiple async requests sure, it's the developer, but previous messages were...

J

James•1/23/25, 6:14 PM

and they could be running at the "same time" if async and the alarm execution runs past whats expected, my understanding is the alarm can only have one scheduled at a time, but once fired, one could schedule it again inside the same execution, or another execution depending on how the async calls are interleaved

Aalgads OK, let me make sure I understand this. If I set a DO to invoke an alarm, the a...

森

森@VerseLink (RW)OP•1/23/25, 8:10 PM

You are overthinking it, it's simple, alarm like other functions are simply like a I/O event like fetch, but triggered by Cloudflare's internal system once the time is to invoke the alarm function.

森

森@VerseLink (RW)OP•1/23/25, 8:12 PM

so it's just like calling multiple fetch, except in this case it's just alarm, and also it automatically retries to call the alarm function just in case it threw an error

森

森@VerseLink (RW)OP•1/23/25, 8:15 PM

so no, they technically don't get "queued", they just get called whenever the time is up, regardless if any alaram or fetch is being invoked or not; this makes them possible to run concurrently

森

森@VerseLink (RW)OP•1/23/25, 8:16 PM

so if you really want to have a "queued" up behaviour, you should simply

async alarm() {
  doSomething();
  await this.ctx.storage.setAlarm(); // set the next alarm 
}

async alarm() {
  doSomething();
  await this.ctx.storage.setAlarm(); // set the next alarm 
}

async alarm() {
  doSomething();
  await this.ctx.storage.setAlarm(); // set the next alarm 
}

async alarm() {
  doSomething();
  await this.ctx.storage.setAlarm(); // set the next alarm 
}

this way the alarm will behave as if they were in queue, and have this nice added bonus that only successful alarm calls will proceed to the next alarm call

森

森@VerseLink (RW)OP•1/23/25, 8:18 PM

before workflow was a thing, you would implement a queue using DO's KV API, and deque one action out, execute it via alarm, and queue for the next action (pretty sure this is also what is happening underneath the hood, just not in JS land)

P

peachneo•1/24/25, 7:45 AM

I have this function:

async function exhaustAlarms(stub: DurableObjectStub) {
    let count = 0;
    while (true) {
        let ran = await runDurableObjectAlarm(stub)    
        // await new Promise(resolve => setTimeout(resolve, 100))
        if (!ran) {
            break
        };
        count++;
    }
    return count;
}

async function exhaustAlarms(stub: DurableObjectStub) {
    let count = 0;
    while (true) {
        let ran = await runDurableObjectAlarm(stub)    
        // await new Promise(resolve => setTimeout(resolve, 100))
        if (!ran) {
            break
        };
        count++;
    }
    return count;
}

async function exhaustAlarms(stub: DurableObjectStub) {
    let count = 0;
    while (true) {
        let ran = await runDurableObjectAlarm(stub)    
        // await new Promise(resolve => setTimeout(resolve, 100))
        if (!ran) {
            break
        };
        count++;
    }
    return count;
}

async function exhaustAlarms(stub: DurableObjectStub) {
    let count = 0;
    while (true) {
        let ran = await runDurableObjectAlarm(stub)    
        // await new Promise(resolve => setTimeout(resolve, 100))
        if (!ran) {
            break
        };
        count++;
    }
    return count;
}

If I don't add some small delay (see the setTimeout), I get this issue:

workerd/io/worker.c++:2082: info: uncaught exception; source = Uncaught (in promise); stack = Error: Durable Object storage is no longer accessible.

P

peachneo•1/24/25, 8:03 AM

overall, testing of DOs seems really fragile...I get inconsistent errors, different errors on live reload than hard restart etc

Ppeachneo I have this function: ``` async function exhaustAlarms(stub: DurableObjectStub)...

森

森@VerseLink (RW)OP•1/24/25, 9:01 AM

What is runDurableObjectAlarm

Ppeachneo overall, testing of DOs seems really fragile...I get inconsistent errors, differ...

森

森@VerseLink (RW)OP•1/24/25, 9:03 AM

I realized this ago, not fragile, but just the error thrown usually doesn’t align with the actual error

L

LearnNMake•1/25/25, 2:46 AM

I am researching migrating to a D1 Database. I need one source of truth for the relational data. I will be updating the database with a backend application. The frontend will be performing reads. I need everyone in the world to get the most up to date data. I read that D1 performs replication geographically so this doesn't seem like a good fit. I would rather there be a delay than a client get the wrong or outdated data. Is it possible to somehow force one persistent storage to be used world wide? Are durable objects a solution for this dilemma? Can I enforce everyone in the United States to somehow use one specific D1 Database?

LLearnNMake I am researching migrating to a D1 Database. I need one source of truth for the ...

B

boop•1/25/25, 8:24 AM

There aren't replicas on D1 yet

LLearnNMake I am researching migrating to a D1 Database. I need one source of truth for the ...

L

lambrospetrou•1/25/25, 10:39 AM

Can I enforce everyone in the United States to somehow use one specific D1 Database?

This applies both to D1 and DO. If you want to use different databases per region, you can create a few of them, and then use your worker request location (in the request headers) to decide where to route your queries (choosing the database/DO that you want). Needs a bit of code, but doable.

I read that D1 performs replication geographically so this doesn't seem like a good fit. I would rather there be a delay than a client get the wrong or outdated data.

Once D1 launches read replication you will still be able to get the latest data if needed, we will not break the default consistency. This would mean though that all your requests would need to reach the primary database (transparent to your code, we would do this for you probably), so you would basically use D1 as its current form forgoing the replication features.

More on this once it's ready and out though, replication is not released yet, so now any query you make to D1 goes to the same database, hence you always get the latest data.

森

森@VerseLink (RW)OP•1/25/25, 11:41 AM

is it faster to store a user's durable id (calculated from the user id) in D1, then query D1 to get the DO object faster; or is it faster to just use the user's id to get the DO?

森

森@VerseLink (RW)OP•1/25/25, 11:42 AM

and it is a good idea the cache the DO's id (store on client via JWT) or just get the DO via the user id everytime instead?

森森@VerseLink (RW)is it faster to store a user's durable id (calculated from the user id) in D1, t...

L

lambrospetrou•1/25/25, 1:36 PM

Definitely directly to the DO. If you go to D1 first, you are already reaching a remote database, so you might as well go to the DO directly.
Read https://developers.cloudflare.com/reference-architecture/diagrams/storage/durable-object-control-data-plane-pattern/ to get some more insight.

森森@VerseLink (RW)and it is a good idea the cache the DO's id (store on client via JWT) or just ge...

L

lambrospetrou•1/25/25, 1:38 PM

There shouldn't be a noticeable difference between these once you have regular traffic. Using

idFromName

idFromName

has one extra hop the first time you query it but then it's cached in cloudflare's datacenters, so assuming you have regular requests, there shoudn't be a performance benefit caching the DO ID yourself.
Always benchmark though, and it always depends on your traffic. Don't optimize prematurely.

森

森@VerseLink (RW)OP•1/25/25, 1:42 PM

Thank you very much!

Let's say we allow users to look up any user's public metadata (for example

/user/{userid}

/user/{userid}

, would it be better if we first look up D1 to see if the user id exists, or directly get the user's DO via

idFromName()

idFromName()

, and see if it was previously initialized, destroy it (via

deleteAll()

deleteAll()

) if not?

I would of course benchmark this, but would like some pointers.

森森@VerseLink (RW)Thank you very much! Let's say we allow users to look up any user's public meta...

L

lambrospetrou•1/25/25, 1:50 PM

and see if it was previously initialized, destroy it (via deleteAll()) if not?

Why would you need to delete all if nothing has been initialized?
In this case you need to make your constructor readonly, not writing anything to storage unless needed.
If you do all your table initialization in the constructor, then this won't work nicely.

Anyway, it depends on your usecase. There is a similar discussion a few weeks ago in this group about initialization done in constructor or not for publicly accessible links.

森

森@VerseLink (RW)OP•1/25/25, 2:19 PM

Got it, looking thru the documentation you provided, I have better understanding of what's happening behind the scene for DO.
Readonly as in it's acceptable to do

SELECT * FROM 'tablename'

SELECT * FROM 'tablename'

and we know the DO is not initialized if the table name doesn't exist? (or querying

sqlite_master

sqlite_master

to check if table exists)

森森@VerseLink (RW)Got it, looking thru the documentation you provided, I have better understanding...

J

James•1/25/25, 4:14 PM

DO object / stub initialization and table initialization are completely independent concepts. The constructor talk is just a convenience some developers take on.

You’ll always get errors querying tables don’t exist until you create them. How you manage migrations including table init is your call.

I’ve been thinking, the total account level limit for DO is 50GB, but D1 has limit of 250 GB; it was

Similar Threads

Similar Threads

Similar Threads