Can I have a DO backed by both SQL and KV? I want to use KV storage for some data and SQL for other

MMozzy Can I have a DO backed by both SQL and KV? I want to use KV storage for some dat...

C

Chaika•2/12/25, 12:11 AM

No, but if you use SQL, the KV APIs still work, same api, and just use a hidden table

D

DanTheGoodman•2/12/25, 12:13 AM

Can you make it such that tests wipe previous DO storage data locally when starting the test? Or delete on exit?

CChaika No, but if you use SQL, the KV APIs still work, same api, and just use a hidden ...

M

MozzyOP•2/12/25, 12:13 AM

neat

DDanTheGoodman Can you make it such that tests wipe previous DO storage data locally when start...

B

Burrito•2/12/25, 1:24 AM

If you are using #vitest-integration-beta, there's an

isolatedStorage

isolatedStorage

option, which IIRC is on by default.

BBurrito If you are using #vitest-integration-beta, there's an `isolatedStorage` option, ...

D

DanTheGoodman•2/12/25, 1:34 AM

maybe not the beta, im def runnign into issues. Also where does it save to so i can delete it?

B

Burrito•2/12/25, 1:34 AM

Well, that answer is assuming you are using Vitest integration. If you are using something else then I'm not sure.

Llambrospetrou Obviously it depends on your traffic, but this is a classic example of how hot p...

M

makrdev•2/12/25, 1:05 PM

I was planning to use a single DO per slug to route traffic to different variations but after I saw your comment I’ve realized that its not a good pattern.

How can we architect such a system like this with DOs then?

Mmakrdev I was planning to use a single DO per slug to route traffic to different variati...

L

lambrospetrou•2/12/25, 1:10 PM

As always it depends on volume. If you have a few hundred requests per second then it should be fine either way.
Once you have thousands per second per DO it's where you need to start sharding more. There is no "general pattern" that works for everywhere. Different scales need different solutions. And obviously complexity scales as well.
I just mentioned that case because for example if something like Prime Video designed 1 DO per show, then at that scale it wouldn't work. If a local radio station did it, probably all is fine.

Llambrospetrou As always it depends on volume. If you have a few hundred requests per second th...

M

makrdev•2/12/25, 3:26 PM

Gotcha, I think I can start with the basic solution(to avoid pre-mature optimization since 500 rps is good enough for now) and then introduce sharding. Thanks

M

MozzyOP•2/12/25, 3:39 PM

Is there any good way to gain insights into a DO's SQL data through a UI? Kind of like D1 has

MMozzy Is there any good way to gain insights into a DO's SQL data through a UI? Kind o...

B

Brayden•2/12/25, 6:39 PM

You can check out https://github.com/outerbase/starbasedb which has some prebuilt components that get deployed with the DO including a visual SQL & table editor. You can also connect your DO then to outerbase.com and use the tools there to visualize and modify your data too.

BBrayden You can check out https://github.com/outerbase/starbasedb which has some prebuil...

M

MozzyOP•2/12/25, 7:50 PM

Sounds cool, I'll check it out

BBrayden You can check out https://github.com/outerbase/starbasedb which has some prebuil...

J

Jehu•2/12/25, 11:06 PM

Does this work with a pre-existing DO?

JJehu Does this work with a pre-existing DO?

B

Brayden•2/12/25, 11:15 PM

I haven’t tried but I believe you should be able to bind it to an existing DO no problems

Hhalftome I'm trying to use websockets with my DO. In the tutorials it invokes the DO via ...

H

Hard@Work•2/13/25, 9:51 AM

Nope, at the moment, WebSockets need to be passed via

fetch

fetch

Hhalftome In my use case I have the option of using either websockets or SSE to periodical...

H

Hard@Work•2/13/25, 10:19 AM

Nope, CPU Time is only when actively processing something. Just keeping a connection open uses no CPU

Hhalftome okay thanks. It will probably have to be backed by some kind of pubsub, and sinc...

H

Hard@Work•2/13/25, 10:53 AM

Yes, though I'm not entirely sure how Redis handles connections, it might make some traffic, even when not actively sending data

G

geg•2/13/25, 11:30 AM

ye u need websocket hibernation

G

geg•2/13/25, 11:31 AM

or make requests on an interval

HHard@Work Nope, CPU Time is only when *actively* processing something. Just keeping a conn...

L

Larry•2/13/25, 8:36 PM

I thought DOs were charged on clock-time, not CPU time. Regarless, @halftome, SSE is not a good choice. DOs can be stood up and torn down any time. Same applies for Workers/Page . Since the browser can't directly access DO (except through a WebSocket connection that was previously made via a Worker), you would need to keep both hops open continuously. That's inneficient from both a technical as well as financial perspective. As @egg said, you want websocket hibernation.

LLarry I thought DOs were charged on clock-time, not CPU time. Regarless, @halftome, S...

H

Hello, I’m Allie!•2/13/25, 8:55 PM

At least if I was understanding correctly, they wanted to terminate the SSE connection within a Worker

H

Hello, I’m Allie!•2/13/25, 8:55 PM

It was just if it was a webSocket that they would choose DOs

H

Hello, I’m Allie!•2/13/25, 8:55 PM

Might be wrong tho

BBrayden I haven’t tried but I believe you should be able to bind it to an existing DO no...

D

DiamondDragon•2/14/25, 6:27 AM

i'd be curious to learn more about this as im trying to get up to speed on using DOs, i had a question here before wondering how people are inspecting the data in their DO instances. Signed up for outbase to quickly check out and I see the D1 connections but not finding how to connect to a DO

1

1984 Ford Laser•2/14/25, 8:18 AM

What's the best way to parallelize calls to multiple DO stubs? I want a Worker to iterate over a list of

user_id

user_id

s and get a stub for each string, and read some data, and collect it together, but want to try and speed that process up as much as possible

11984 Ford Laser What's the best way to parallelize calls to multiple DO stubs? I want a Worker t...

L

lambrospetrou•2/14/25, 8:37 AM

Do parallel calls without awaiting them and wait at the end or after batches of N. https://www.npmjs.com/package/p-queue/v/6.0.2 is a popular package used for more control ovr the concurrency.

Llambrospetrou Do parallel calls without awaiting them and wait at the end or after batches of ...

1

1984 Ford Laser•2/14/25, 8:38 AM

exactly what I'm looking for thank you

DDiamondDragon i'd be curious to learn more about this as im trying to get up to speed on using...

B

Brayden•2/14/25, 12:59 PM

You need to bind your existing DO to a StarbaseDB Worker (clone this & update the wrangler.toml to point to your DO and deploy it to CF – https://github.com/outerbase/starbasedb) then you can connect to Outerbase for example with the "Starbase" option. Edit: made this how-to article if it helps https://x.com/BraydenWilmoth/status/1890408856892629313

CChaika No, but if you use SQL, the KV APIs still work, same api, and just use a hidden ...

M

Matt•2/14/25, 2:22 PM

That's good to know. So by enabling sqlite for the DO it no longer uses the KV and it's all sqlite? So the rules for optimizing KV with in-memory caching and skipping await for puts to rely on the output gate would no longer apply and everything is synchronous? And does usage/pricing switch over to the sqlite?

M

Matt•2/14/25, 2:31 PM

Well, I guess the read in-memory caching would probably still exist, but can you now use get without await?

MMatt Well, I guess the read in-memory caching would probably still exist, but can you...

L

lambrospetrou•2/14/25, 3:57 PM

The KV API is the same, so still async and I suggest keep your awaits to get good IDE intellisense, regardless how the underlying storage works.

P

plgingras88•2/14/25, 7:33 PM

Hey pals, I am new to DO. I have been amazed until now. Quick question : if I delete a DO instance, will it be recreated potentially elsewhere or the location is written in the stone for good ? Thank you !!

Pplgingras88 Hey pals, I am new to DO. I have been amazed until now. Quick question : if I de...

L

lambrospetrou•2/14/25, 7:50 PM

Mostly depends on the underlying Cloudflare infrastructure. It's possible to move, and they do move over periods of time. Still within the same region (location hint) though. Also, this is transparent to you and your code.

P

peachneo•2/14/25, 11:34 PM

are there any fleshed out examples of testing DO alarms with vitest?

Ppeachneo are there any fleshed out examples of testing DO alarms with vitest?

L

lambrospetrou•2/14/25, 11:47 PM

https://github.com/cloudflare/workers-sdk/blob/main/fixtures/vitest-pool-workers-examples/durable-objects/test/alarm.test.ts

P

peachneo•2/15/25, 1:31 AM

I get some strange leakage between tests...

I have a test that succeeds if other tests are commented out, or if it's anywhere but the first test to run. It will also succeed on live reload.

It fails because

const ran = await runDurableObjectAlarm(stub); expect(ran).toBe(true);

const ran = await runDurableObjectAlarm(stub); expect(ran).toBe(true);

const ran = await runDurableObjectAlarm(stub); expect(ran).toBe(true);

const ran = await runDurableObjectAlarm(stub); expect(ran).toBe(true);// expects true but is false

    it("adds jobs and schedules an alarm", async () => {
      const result = await runInDurableObject(stub, async (instance, state) => {
        return await instance.addJobs([
          { url: "https://example.com" },
          { url: "https://example.org" },
        ]);
      });
      expect(result).toBe("Jobs added");

      // Run the alarm (which calls processQueue & processTimeouts)
      const ran = await runDurableObjectAlarm(stub);
      expect(ran).toBe(true);

      const status = await runInDurableObject(stub, async (instance) => {
        return await instance.getStatus();
      });
      expect(status).toEqual([{ status: "pending", count: 2 }]);
    });

    it("adds jobs and schedules an alarm", async () => {
      const result = await runInDurableObject(stub, async (instance, state) => {
        return await instance.addJobs([
          { url: "https://example.com" },
          { url: "https://example.org" },
        ]);
      });
      expect(result).toBe("Jobs added");

      // Run the alarm (which calls processQueue & processTimeouts)
      const ran = await runDurableObjectAlarm(stub);
      expect(ran).toBe(true);

      const status = await runInDurableObject(stub, async (instance) => {
        return await instance.getStatus();
      });
      expect(status).toEqual([{ status: "pending", count: 2 }]);
    });

    it("adds jobs and schedules an alarm", async () => {
      const result = await runInDurableObject(stub, async (instance, state) => {
        return await instance.addJobs([
          { url: "https://example.com" },
          { url: "https://example.org" },
        ]);
      });
      expect(result).toBe("Jobs added");

      // Run the alarm (which calls processQueue & processTimeouts)
      const ran = await runDurableObjectAlarm(stub);
      expect(ran).toBe(true);

      const status = await runInDurableObject(stub, async (instance) => {
        return await instance.getStatus();
      });
      expect(status).toEqual([{ status: "pending", count: 2 }]);
    });

    it("adds jobs and schedules an alarm", async () => {
      const result = await runInDurableObject(stub, async (instance, state) => {
        return await instance.addJobs([
          { url: "https://example.com" },
          { url: "https://example.org" },
        ]);
      });
      expect(result).toBe("Jobs added");

      // Run the alarm (which calls processQueue & processTimeouts)
      const ran = await runDurableObjectAlarm(stub);
      expect(ran).toBe(true);

      const status = await runInDurableObject(stub, async (instance) => {
        return await instance.getStatus();
      });
      expect(status).toEqual([{ status: "pending", count: 2 }]);
    });

The add jobs instance call simply inserts some records and sets an alarm

P

peachneo•2/15/25, 1:38 AM

I know there's this in the docs

Durable Object alarms are not reset between test runs and do not respect isolated storage. Ensure you delete or run all alarms with runDurableObjectAlarm() scheduled in each test before finishing the test.

Durable Object alarms are not reset between test runs and do not respect isolated storage. Ensure you delete or run all alarms with runDurableObjectAlarm() scheduled in each test before finishing the test.

Durable Object alarms are not reset between test runs and do not respect isolated storage. Ensure you delete or run all alarms with runDurableObjectAlarm() scheduled in each test before finishing the test.

Durable Object alarms are not reset between test runs and do not respect isolated storage. Ensure you delete or run all alarms with runDurableObjectAlarm() scheduled in each test before finishing the test.

But I believe my afterEach should cover this

  afterEach(async () => {
    // Clear any pending alarms between tests.
    await runInDurableObject(stub, async (_instance, state) => {
      await state.storage.deleteAlarm();
    });
    vi.restoreAllMocks();
  });

  afterEach(async () => {
    // Clear any pending alarms between tests.
    await runInDurableObject(stub, async (_instance, state) => {
      await state.storage.deleteAlarm();
    });
    vi.restoreAllMocks();
  });

  afterEach(async () => {
    // Clear any pending alarms between tests.
    await runInDurableObject(stub, async (_instance, state) => {
      await state.storage.deleteAlarm();
    });
    vi.restoreAllMocks();
  });

  afterEach(async () => {
    // Clear any pending alarms between tests.
    await runInDurableObject(stub, async (_instance, state) => {
      await state.storage.deleteAlarm();
    });
    vi.restoreAllMocks();
  });

1

1984 Ford Laser•2/15/25, 8:42 AM

If I have a Worker that iterates over a list of strings, gets a stub for each string, then calls

this.state.getWebSockets

this.state.getWebSockets

this.state.getWebSockets

this.state.getWebSockets on each stub, is this 1 DO request per stub? Do these count toward the subrequest limit of the initially invoked Worker?

S

sneakyf1shy•2/15/25, 3:38 PM

is there a proper way to use durable objects inside NextJS on cloudflare pages?

S

sneakyf1shy•2/15/25, 3:38 PM

I can't seem to find anything new online, only like a github issue closed in 2023

Can I have a DO backed by both SQL and KV? I want to use KV storage for some data and SQL for other

Similar Threads

Similar Threads

Similar Threads