Website says 1000 rps under limits somewhere and there is a DOG service that allows for scaling DO h

Website says 1000 rps under limits somewhere and there is a DOG service that allows for scaling DO horizontally - with DOs having 128mb memory to play with it wouldn’t be able to handle tens of thousands.

https://developers.cloudflare.com/durable-objects/platform/limits/

I have leaned into this project to help me scale to 10k:

https://github.com/cloudflare/dog

Cloudflare Docs

Limits · Cloudflare Durable Objects docs

Durable Objects are only available on the Workers Paid plan. Durable Objects limits are the same as Workers Limits, as well as the following limits that are specific to Durable Objects:

GitHub

GitHub - cloudflare/dog: Durable Object Groups

Durable Object Groups. Contribute to cloudflare/dog development by creating an account on GitHub.

CCodingCoop | Co-Founder Nullshot Website says 1000 rps under limits somewhere and there is a DOG service that all...

mihaaai•3/9/25, 11:02 AM

The 1000rps is for requests, not websocket messages

AAshley Someone asked me the other day, and now I’m curious, what’s the max number of co...

mihaaai•3/9/25, 11:06 AM

With hibernation the DO server should be able to hold up to 32k client connections at a time and broadcast between each-others

DaniFoldi•3/9/25, 11:01 PM

although if you're broadcasting to all of them, from my experience it can't really handle more than a few thousand, so I'd recommend soft limiting to 2-4k

DaniFoldi•3/9/25, 11:02 PM

and at around 20-25k connections it became pretty unreliable

DaniFoldi•3/9/25, 11:02 PM

(these are from my testing done a few months ago)

DDaniFoldi although if you're broadcasting to all of them, from my experience it can't real...

akazwz•3/10/25, 3:33 PM

single Do right ?

Aakazwz single Do right ?

DaniFoldi•3/10/25, 3:33 PM

correct

rh•3/12/25, 10:45 AM

Is something like https://github.com/cloudflare/workers-rs also available for C++? Or does anybody have an example on how to use the SQLite storage API in a C++ to WASM build?

GitHub

GitHub - cloudflare/workers-rs: Write Cloudflare Workers in 100% Ru...

Write Cloudflare Workers in 100% Rust via WebAssembly - cloudflare/workers-rs

Larry•3/13/25, 4:09 PM

Is there any way to know the "name" as in "foo" in this example,

env.MY_DURABLE_OBJECT.idFromName("foo")

env.MY_DURABLE_OBJECT.idFromName("foo")

, once inside of a DO instance? At one point I know the answer used to be "no" but I'm wondering if this has changed.

LLarry Is there any way to know the "name" as in "foo" in this example, `env.MY_DURABLE...

Unsmart•3/13/25, 4:12 PM

You would have to send the name to the DO yourself and save it

LLarry Is there any way to know the "name" as in "foo" in this example, `env.MY_DURABLE...

Hard@Work•3/13/25, 4:17 PM

Here's the thread: https://github.com/cloudflare/workerd/issues/2240

GitHub

bug: `DurableObjectId::id.name` doesn't work · Issue #2240 · cloudf...

(maybe this is a feature request?) When creating a durable object with .idFromName(name), I would expect this.ctx.id.name inside a durable object. to reflect the same. The types imply that it shoul...

Iggy•3/13/25, 6:35 PM

what's the SQL-in-DO durability? is it the same as R2 with 11 9s?

IIggy what's the SQL-in-DO durability? is it the same as R2 with 11 9s?

lambrospetrou•3/13/25, 6:52 PM

No numbers are advertised, but all writes are acknowledged by a quorum from five followers before returning to your worker. See details in https://blog.cloudflare.com/sqlite-in-durable-objects/

To solve this problem, we take advantage of our global network. Every time SQLite commits a transaction, SRS will immediately forward the change log to five "follower" machines across our network. Once at least three of these followers respond that they have received the change, SRS informs the application that the write is confirmed. (As discussed earlier, the write confirmation opens the Durable Object's "output gate", unblocking network communications to the rest of the world.)

When a follower receives a change, it temporarily stores it in a buffer on local disk, and then awaits further instructions. Later on, once SRS has successfully uploaded the change to object storage as part of a batch, it informs each follower that the change has been persisted. At that point, the follower can simply delete the change from its buffer.
[...]
Each of a database's five followers is located in a different physical data center. Cloudflare's network consists of hundreds of data centers around the world, which means it is always easy for us to find four other data centers nearby any Durable Object (in addition to the one it is running in). In order for a confirmed write to be lost, then, at least four different machines in at least three different physical buildings would have to fail simultaneously (three of the five followers, plus the Durable Object's host machine). Of course, anything can happen, but this is exceedingly unlikely.

Iggy•3/13/25, 6:54 PM

awesome, thank you!

Llambrospetrou No numbers are advertised, but all writes are acknowledged by a quorum from five...

Iggy•3/13/25, 7:39 PM

in the case of a DO w/ SQLite becoming unresponsive and the followers have to start a new one, do you know the details here? does it have to replicate the SQLite back to the new DO's disk, or does it use the same disk, or something else?

IIggy in the case of a DO w/ SQLite becoming unresponsive and the followers have to st...

lambrospetrou•3/13/25, 7:43 PM

If the new DO ends up on the same machine, then the same disk is used. If it ends up in a separate machine, it loads the db from cold storage and resumes replication from that point on.

Iggy•3/13/25, 7:44 PM

do you know the likelihood of being on the same machine vs a new one? or how frequently do DOs become unresponsive at the infra level?

IIggy do you know the likelihood of being on the same machine vs a new one? or how fre...

Milan•3/13/25, 8:34 PM

Generally it should be pretty likely it goes back to the same machine, unless that machine becomes unhealthy (in which case you wouldn't even want it to be scheduled there again, a cold start is preferable to no start!)

It's really tricky to say how often individual DOs suffer from infra issues:
1. not all PoPs are the same,
2. you could have some DOs in a namespace have nearly perfect availability over some period, and then a handful with terrible availability (ex. if they only do 10 requests in a month, but all 10 were during an incident, or when a PoP was under a lot of load and we were doing maintenance so there were fewer resources than normal, etc)

Not that any of this has any bearing on durability/application correctness

Iggy•3/13/25, 8:37 PM

makes sense, thanks! Does CF have an uptime SLA? I couldn’t find one

IIggy makes sense, thanks! Does CF have an uptime SLA? I couldn’t find one

1984 Ford Laser•3/13/25, 9:00 PM

100%, with it being possible to apply for a credit for the amount of time that was lost X your average usage. Usually, large platform downtimes are measured in minutes, not hours, so even if there is possibility for a credit, it wouldn't even be that much

Iggy•3/13/25, 9:02 PM

yeah I’m familiar with the contracts but do they actually advertise 100% uptime?

generally it’s a proxy for stability, idrc about the credit aspect

IIggy yeah I’m familiar with the contracts but do they actually advertise 100% uptime?...

Isaac McFadyen•3/13/25, 9:03 PM

Because the service isn't different for Enterprise versus non-Enterprise, the uptime for customers on SLA and not on SLA is likely to be identical. But yes, Enterprise customers have 100% uptime SLAs, if they request it.

1984 Ford Laser•3/13/25, 9:09 PM

oh yeah I forgot the Ent requirement there. Stability is very good here, I think there's been like maybe 2 hours of platform downtime in the last 6 months or something, and that was technically a subset of products, not the entire thing

11984 Ford Laser oh yeah I forgot the Ent requirement there. Stability is very good here, I think...

Iggy•3/13/25, 9:20 PM

isn’t that like 99.95%?

IIggy isn’t that like 99.95%?

1984 Ford Laser•3/13/25, 9:29 PM

On that non-specific timeframe I provided, maybe. Not sure why you're so concerned about possible downtime, go build some stuff!

Iggy•3/13/25, 9:30 PM

I’m doing service provider evaluation, I can’t not be concerned about possible downtime

Iggy•3/13/25, 9:31 PM

historical downtime and failover behavior are important

1984 Ford Laser•3/13/25, 9:48 PM

Hopefully others can chime in here, but in my experience CF works very hard to provide constant service, and when there is a downtime, it is a followed with an in-depth post-mortem explaining what happened. Of course every platform has downtime, but part of the benefit of the global network is that its very hard to take down the entire network/platform, one location fails then we just spin up another instance elsewhere. Durable Objects themselves don't really have platform downtime, I can't remember any in the last 6 months I've been working with it

1984 Ford Laser•3/13/25, 9:49 PM

https://blog.cloudflare.com/cloudflare-incident-on-february-6-2025/
https://blog.cloudflare.com/cloudflare-incident-on-november-14-2024-resulting-in-lost-logs/

Here are some incident post-mortems if you're curious. Can help you to understand what errors may occur, and how they are handled internally and from an outward perspective

Iggy•3/13/25, 9:56 PM

Thanks! CF makes amazing public posts

IIggy Thanks! CF makes amazing public posts

1984 Ford Laser•3/13/25, 9:57 PM

Very transparent, has given me a lot of confidence in the platform. The devs are also super clever and have built some really cool tech between everything they do

o7•3/15/25, 6:27 AM

is the SQLite in DO usage separate than D1? 50B (25b + 25b) reads combined for a paid Worker plan?

o7•3/15/25, 6:28 AM

I’m assuming not but must confirm

Oo7 is the SQLite in DO usage separate than D1? 50B (25b + 25b) reads combined for a...

lambrospetrou•3/15/25, 10:00 AM

separate

o7•3/15/25, 4:35 PM

woah

o7•3/15/25, 4:35 PM

thats awesome, thanks

plgingras88•3/16/25, 12:37 PM

I am experimenting with DO SQLite and it's a real charm so far ! I however can't find a clear answer on how to use SQL transactions with SQLite and if it may have a signifiant impact on performance for tables with few thousands of rows. A user may insert or update up to 200 rows at the same time.

Since it's BETA, there is still no clear example in the docs for SQLite
- https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction

And a tutorial that inserts 60 rows says : "Since SQLite in DOs is fast, we can do a query per INSERT instead of batching them in a transaction."
- https://developers.cloudflare.com/durable-objects/tutorials/build-a-seat-booking-app/#3-create-table-for-each-flight

Pplgingras88 I am experimenting with DO SQLite and it's a real charm so far ! I however can't...

lambrospetrou•3/16/25, 1:27 PM

I however can't find a clear answer on how to use SQL transactions with SQLite and if it may have a signifiant impact on performance for tables with few thousands of rows

So, is performance of transactions your question?
SQLite DOs are single threaded anyway, so if you will make 200 inserts separately or 1 transaction containing the 200 inserts should have similar performance. The difference is if you have other JS async code in between those inserts.

Within a single transaction all your inserts will be atomic, whereas if you do some and then do async IO (e.g.

fetch

fetch

) and then the rest, your DO could be processing other requests before returning to do the remaining inserts.
If you don't have any async code, then the DO runtime will probably also just do all the inserts in the same (implicit) transaction.

Llambrospetrou > I however can't find a clear answer on how to use SQL transactions with SQLite...

plgingras88•3/16/25, 2:51 PM

Thanks, this indeed answer the perf question. The second question was how to proceed with a transaction using SQLite and for example rollback somewhere in the middle ?

Pplgingras88 Thanks, this indeed answer the perf question. The second question was how to pro...

lambrospetrou•3/16/25, 3:31 PM

You can use https://developers.cloudflare.com/durable-objects/api/storage-api/#transaction and throw from within when you want to not apply it. If you prefer a synchronous version you can use https://developers.cloudflare.com/durable-objects/api/storage-api/#transactionsync as well.

Using any

sql.exec

sql.exec

operation within the closure of the above methods will run all those queries within the same transaction, and throwing within that closure is similar to rolling back.

plgingras88•3/16/25, 3:49 PM

Thank you, very much appreciated!

fforres•3/18/25, 12:26 PM

How do you actually debug a

Durable Object's isolate exceeded its memory limit and was reset.

Durable Object's isolate exceeded its memory limit and was reset.

error ?

Is there a way to profile an isolate on cloudflare's infra?

fforres•3/18/25, 12:58 PM

local profiling it is

I come back to that question every few months xD
Would be great if there was. Given the scale of workers, imagine quite hard

stephen•3/18/25, 4:03 PM

running into issue with deployments of durable objects, we setup 2 envs with diff wrangler files (

wrangler.{env).json

wrangler.{env).json

) both deployed fine in CI (worker+a few durable objects)

after initiating a second deploy though im getting this:

Cannot apply new-sqlite-class migration to class 'MyObject' that is already depended on by existing Durable Objects [code: 10074]

Cannot apply new-sqlite-class migration to class 'MyObject' that is already depended on by existing Durable Objects [code: 10074]

I had a look through this discord and it seems like this issue was mentioned a couple times with same error code but couldnt see a clear answer on resolving it from prev messages on my search

would be great to get some guidance here, snippet of wrangler is below:

/**
 * For more details on how to configure Wrangler, refer to:
 * https://developers.cloudflare.com/workers/wrangler/configuration/
 */
{
    "$schema": "node_modules/wrangler/config-schema.json",
    "name": "my-app",
    "main": "src/index.ts",
    "compatibility_date": "2025-02-24",
    "compatibility_flags": [
        "nodejs_compat"
    ],
    "observability": {
        "enabled": true
    },
    "durable_objects": {
        "bindings": [
      {
              "name": "Brain",
                "class_name": "Brain"
            },
      {
              "name": "Chat",
                "class_name": "Chat"
            },        
    ]
    },
    "migrations": [
    {
            "tag": "v1",
            "new_sqlite_classes": [
                "Brain"
            ]
        },
    {
            "tag": "v1",
            "new_sqlite_classes": [
                "Chat"
            ]
        }
}

/**
 * For more details on how to configure Wrangler, refer to:
 * https://developers.cloudflare.com/workers/wrangler/configuration/
 */
{
    "$schema": "node_modules/wrangler/config-schema.json",
    "name": "my-app",
    "main": "src/index.ts",
    "compatibility_date": "2025-02-24",
    "compatibility_flags": [
        "nodejs_compat"
    ],
    "observability": {
        "enabled": true
    },
    "durable_objects": {
        "bindings": [
      {
              "name": "Brain",
                "class_name": "Brain"
            },
      {
              "name": "Chat",
                "class_name": "Chat"
            },        
    ]
    },
    "migrations": [
    {
            "tag": "v1",
            "new_sqlite_classes": [
                "Brain"
            ]
        },
    {
            "tag": "v1",
            "new_sqlite_classes": [
                "Chat"
            ]
        }
}

Thanks for any help

Sstephen running into issue with deployments of durable objects, we setup 2 envs with dif...

boop•3/18/25, 4:49 PM

/**
 * For more details on how to configure Wrangler, refer to:
 * https://developers.cloudflare.com/workers/wrangler/configuration/
 */
{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "my-app",
  "main": "src/index.ts",
  "compatibility_date": "2025-02-24",
  "compatibility_flags": ["nodejs_compat"],
  "observability": {
    "enabled": true
  },
  "durable_objects": {
    "bindings": [
      {
        "name": "Brain",
        "class_name": "Brain"
      },
      {
        "name": "Chat",
        "class_name": "Chat"
      }
    ]
  },
  "migrations": [
    {
      "tag": "v1",
      "new_sqlite_classes": ["Brain", "Chat"]
    }
  ]
}

/**
 * For more details on how to configure Wrangler, refer to:
 * https://developers.cloudflare.com/workers/wrangler/configuration/
 */
{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "my-app",
  "main": "src/index.ts",
  "compatibility_date": "2025-02-24",
  "compatibility_flags": ["nodejs_compat"],
  "observability": {
    "enabled": true
  },
  "durable_objects": {
    "bindings": [
      {
        "name": "Brain",
        "class_name": "Brain"
      },
      {
        "name": "Chat",
        "class_name": "Chat"
      }
    ]
  },
  "migrations": [
    {
      "tag": "v1",
      "new_sqlite_classes": ["Brain", "Chat"]
    }
  ]
}

Might just be a copy error, but you're missing a closing bracket. The migrations array should also only have 1 unique tag per object.

Bboop ```json /** * For more details on how to configure Wrangler, refer to: * https...

stephen•3/18/25, 5:47 PM

that was indeed a copy error - but noted on the second point thank you will try with single migrations array only having 1 unique tag, interesting though that it deployed in first place

stephen•3/18/25, 7:33 PM

odd

stephen•3/18/25, 7:36 PM

nvm actually @boop that seemed to work

stephen•3/18/25, 7:37 PM

boop•3/18/25, 7:38 PM

Glad it was a simple fix

/** * For more details on how to configure Wrangler, refer to: * https://developers.cloudflare.com/workers/wrangler/configuration/ */ { "$schema": "node_modules/wrangler/config-schema.json", "name": "my-app", "main": "src/index.ts", "compatibility_date": "2025-02-24", "compatibility_flags": [ "nodejs_compat" ], "observability": { "enabled": true }, "durable_objects": { "bindings": [ { "name": "Brain", "class_name": "Brain" }, { "name": "Chat", "class_name": "Chat" }, ] }, "migrations": [ { "tag": "v1", "new_sqlite_classes": [ "Brain" ] }, { "tag": "v1", "new_sqlite_classes": [ "Chat" ] } }

/** * For more details on how to configure Wrangler, refer to: * https://developers.cloudflare.com/workers/wrangler/configuration/ */ { "$schema": "node_modules/wrangler/config-schema.json", "name": "my-app", "main": "src/index.ts", "compatibility_date": "2025-02-24", "compatibility_flags": ["nodejs_compat"], "observability": { "enabled": true }, "durable_objects": { "bindings": [ { "name": "Brain", "class_name": "Brain" }, { "name": "Chat", "class_name": "Chat" } ] }, "migrations": [ { "tag": "v1", "new_sqlite_classes": ["Brain", "Chat"] } ] }

Website says 1000 rps under limits somewhere and there is a DOG service that allows for scaling DO h

Similar Threads

Similar Threads

Similar Threads