Focusing just on the Worker code, since the first is mostly Drizzle stuff I guess which I am not fam

Llambrospetrou Focusing just on the Worker code, since the first is mostly Drizzle stuff I gues...

A

Althaf•2/3/25, 11:23 AM

Hey thankyou for the response. I have finally fixed it

Llambrospetrou Focusing just on the Worker code, since the first is mostly Drizzle stuff I gues...

A

Althaf•2/3/25, 11:24 AM

It was bind name mismatch

K

kamama•2/3/25, 1:15 PM

Btw, anyway to create and bind D1 on runtime ?

Kkamama Btw, anyway to create and bind D1 on runtime ?

M

Max (@rozenmd)•2/3/25, 3:42 PM

only via the HTTP API

J

jd•2/3/25, 8:02 PM

hi qq -- im trying to convert a largely kv-based lookup metadata of ~7GB in size of json and gets 10+ GB in size later with parallel reads, updates. Metadata needs persistence storage for consistency and availability too.

My metadata has 10M+ entries where each entry is a small dict of mostly nested strings of a loose schema, mixed with floats.

Is this size a good use case for cloudflare d1 / kv? thanks!

Jjd hi qq -- im trying to convert a largely kv-based lookup metadata of ~7GB in size...

M

Max (@rozenmd)•2/3/25, 8:05 PM

I'd probably pick KV for that, but there's a doc to help you choose: https://developers.cloudflare.com/workers/platform/storage-options/#choose-a-storage-product

Cloudflare Docs

Choosing a data or storage product. · Cloudflare Workers docs

Storage and database options available on Cloudflare's developer platform.

S

sahilmob•2/4/25, 7:53 PM

Hi, I'm building a backend with hono and better-auth, since better auth needs a DB I can only pass the DB instance using I middleware so I wonder if it is ok I initialize better auth only once in cloudflare worker, I mean to create a singleton that lives as long as the worker do. here is an example:

export class Auth {
  static initialized = false;
  private static instance: ReturnType<typeof betterAuth>;

  static async init(c: Context<Env>) {
    if (Auth.instance) {
      await Auth.getSession(c);
      Auth.initialized = true;
      return Object.freeze(Auth.instance);
    }

    Auth.instance = betterAuth({
      database: drizzleAdapter(
        {
          dialect: new D1Dialect({
            database: c.env.DB,
          }),
        },
      ),
    });

    Auth.initialized = true;

    await Auth.getSession(c);

    return Object.freeze(Auth.instance);
  }

  static async getSession(c: Context) {
    const session = await Auth.instance.api.getSession({
      headers: c.req.raw.headers,
    });

    c.set("user", session?.user ?? null);
    c.set("session", session?.session ?? null);

    return session;
  }

  static getInstance() {
    if (!Auth.instance) {
      throw new Error("Auth instance not initialized");
    }

    return Object.freeze(Auth.instance);
  }
}

export class Auth {
  static initialized = false;
  private static instance: ReturnType<typeof betterAuth>;

  static async init(c: Context<Env>) {
    if (Auth.instance) {
      await Auth.getSession(c);
      Auth.initialized = true;
      return Object.freeze(Auth.instance);
    }

    Auth.instance = betterAuth({
      database: drizzleAdapter(
        {
          dialect: new D1Dialect({
            database: c.env.DB,
          }),
        },
      ),
    });

    Auth.initialized = true;

    await Auth.getSession(c);

    return Object.freeze(Auth.instance);
  }

  static async getSession(c: Context) {
    const session = await Auth.instance.api.getSession({
      headers: c.req.raw.headers,
    });

    c.set("user", session?.user ?? null);
    c.set("session", session?.session ?? null);

    return session;
  }

  static getInstance() {
    if (!Auth.instance) {
      throw new Error("Auth instance not initialized");
    }

    return Object.freeze(Auth.instance);
  }
}

export class Auth {
  static initialized = false;
  private static instance: ReturnType<typeof betterAuth>;

  static async init(c: Context<Env>) {
    if (Auth.instance) {
      await Auth.getSession(c);
      Auth.initialized = true;
      return Object.freeze(Auth.instance);
    }

    Auth.instance = betterAuth({
      database: drizzleAdapter(
        {
          dialect: new D1Dialect({
            database: c.env.DB,
          }),
        },
      ),
    });

    Auth.initialized = true;

    await Auth.getSession(c);

    return Object.freeze(Auth.instance);
  }

  static async getSession(c: Context) {
    const session = await Auth.instance.api.getSession({
      headers: c.req.raw.headers,
    });

    c.set("user", session?.user ?? null);
    c.set("session", session?.session ?? null);

    return session;
  }

  static getInstance() {
    if (!Auth.instance) {
      throw new Error("Auth instance not initialized");
    }

    return Object.freeze(Auth.instance);
  }
}

export class Auth {
  static initialized = false;
  private static instance: ReturnType<typeof betterAuth>;

  static async init(c: Context<Env>) {
    if (Auth.instance) {
      await Auth.getSession(c);
      Auth.initialized = true;
      return Object.freeze(Auth.instance);
    }

    Auth.instance = betterAuth({
      database: drizzleAdapter(
        {
          dialect: new D1Dialect({
            database: c.env.DB,
          }),
        },
      ),
    });

    Auth.initialized = true;

    await Auth.getSession(c);

    return Object.freeze(Auth.instance);
  }

  static async getSession(c: Context) {
    const session = await Auth.instance.api.getSession({
      headers: c.req.raw.headers,
    });

    c.set("user", session?.user ?? null);
    c.set("session", session?.session ?? null);

    return session;
  }

  static getInstance() {
    if (!Auth.instance) {
      throw new Error("Auth instance not initialized");
    }

    return Object.freeze(Auth.instance);
  }
}

and in the worker handler:

const app = new Hono<Env>();

app.use("*", async (c, next) => {
  if (Auth.initialized) return next();

  await Auth.init(c);
  return next();
});

app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw));

export default app;
export type ApiRoutes = typeof apiRoutes;

const app = new Hono<Env>();

app.use("*", async (c, next) => {
  if (Auth.initialized) return next();

  await Auth.init(c);
  return next();
});

app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw));

export default app;
export type ApiRoutes = typeof apiRoutes;

const app = new Hono<Env>();

app.use("*", async (c, next) => {
  if (Auth.initialized) return next();

  await Auth.init(c);
  return next();
});

app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw));

export default app;
export type ApiRoutes = typeof apiRoutes;

const app = new Hono<Env>();

app.use("*", async (c, next) => {
  if (Auth.initialized) return next();

  await Auth.init(c);
  return next();
});

app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw));

export default app;
export type ApiRoutes = typeof apiRoutes;

The rational behind this approach is to not reinitialize the auth instance and make a db connection for each request

Ssahilmob Hi, I'm building a backend with hono and better-auth, since better auth needs a ...

L

lambrospetrouOP•2/5/25, 11:02 AM

It's not recommended in general to reuse

env

env

across Worker requests, which is why you receive it in every request.
Also, D1 doesn't have the traditional TCP connection establishing as other databases like Postgres. Every D1 query you make uses our internal networking to reach your remote database anyway, there isn't a D1 connection concept at all, it's similar as calling a service binding/WorkerEntrypoint.

So, my suggestion is to not store env globally and reuse it, otherwise you might be seeing weird behavior occassionally which will be hard to debug.

C

ct_roy•2/5/25, 12:34 PM

Hi folks, i'm seeing 25-30 sec response times on basic inserts into a simple D1 table today. Is there something up with D1 at the moment? i'm also seeing frequent "Error: D1_ERROR: Network connection lost." messages.

K

kake•2/5/25, 4:58 PM

Hey guys, this is my first time using d1 and im getting > 1000ms latencies for simple selects. is this to be expected? ive seen other people having the same issue on the forum aswell but there seems to be no responses. even if i was on the other side of the world to the db the latency should be max 300ms. 1500ms is unacceptable for users in most cases. I have tried to access it from workers aswell with more success (getting down to 800ms) but this is still way too high

K

Khafra•2/5/25, 5:14 PM

Started getting this error and the database is extremely slow:

D1_ERROR: D1 DB is overloaded. Requests queued for too long., stack: Error: D1_ERROR: D1 DB is overloaded. Requests queued for too long.

D1_ERROR: D1 DB is overloaded. Requests queued for too long., stack: Error: D1_ERROR: D1 DB is overloaded. Requests queued for too long.

the error is happening on a batch query. Every column has an index. This started happening out of the blue, but it's not the first time I've had similar issues with d1 (the same database, different tables): workers-and-pages-helpD1 in workers constant query errors & timeouts

Kkake Hey guys, this is my first time using d1 and im getting > 1000ms latencies for s...

L

lambrospetrouOP•2/5/25, 5:17 PM

ive seen other people having the same issue on the forum aswell but there seems to be no responses

It's mentioned many times in the group actually why that is.

The REST API is not how you are meant to be using D1 at the moment as part of your production applications. To get the best performance, you should use the Workers binding and access D1 through your worker. The REST API goes through a centralized location and has different layers adding latency, more aggressive rate limiting, and is therefore not suitable to be used in your actual application to access D1 in a hot path.

There has been recent improvements in the D1 latencies when using the Worker binding and you get pretty much a roundtrip's worth of latency: https://developers.cloudflare.com/d1/platform/changelog/#2025-01-07

Cloudflare Docs

Changelog · Cloudflare D1 docs

Subscribe to RSS

Llambrospetrou > ive seen other people having the same issue on the forum aswell but there seem...

K

kake•2/5/25, 5:19 PM

i see, thank you for your reply

KKhafra Started getting this error and the database is extremely slow: `D1_ERROR: D1 DB ...

K

Khafra•2/5/25, 5:22 PM

bumping so it doesn't get hidden

K

krofdrakula•2/5/25, 7:52 PM

we've just had a massive hit on performance when adding a few new indexes on one of the tables in our D1 database because the query planner used the incorrect index. we pushed out a quick fix by forcing queries to use the old index using

INDEXED BY

INDEXED BY

while i read through the sqlite docs to determine what the correct course of action is.

reading through the query planner docs, i found the chapter on

ANALYZE

ANALYZE

and am wondering whether D1 periodically runs

PRAGMA optimize

PRAGMA optimize

. if not, is this something that should be done by us (the users of D1) periodically or after running migrations that would update tables and add/remove indexes?

Z

zwily•2/6/25, 4:59 AM

is the d1 api struggling right now?

Z

zwily•2/6/25, 5:02 AM

my D1 database has gotten super slow (bound to a worker) too... ugh

Zzwily is the d1 api struggling right now?

M

Max (@rozenmd)•2/6/25, 6:23 AM

just to help us look into this, is it also slow from the worker?

Y

yinxingmaiming6409•2/6/25, 8:28 AM

Is there a problem with R2? Failed to upload images and access.

N

Nlea•2/6/25, 10:28 AM

Hello everyone!

@boots and I were discussing taking D1 with Drizzle to production during our stream yesterday. We are working on a setup and scripts to make the process as easy and seamless as possible. We would love to hear your feedback on our ideas :
https://www.youtube.com/watch?v=4x-AvV5x1mE&t=6s

YouTubeFiberplane

Honc Out #4: Taking D1 Into Production

In this episode of Honc Out, we’re taking D1 into production!

HONC comes with a D1 template that makes it easy to spin up D1 locally for development. However, when deploying a Cloudflare Worker to production, we need to create a D1 instance in Cloudflare and connect it to our deployed worker.

We’ll walk through the configuration steps to ...

Tthomfre I'm getting lots of D1 errors today, in addition to it being very slow. Random e...

M

Max (@rozenmd)•2/6/25, 1:42 PM

We've seen a few reports of this over the past few days, what's your databaseId so we can investigate?

KKhafra Started getting this error and the database is extremely slow: `D1_ERROR: D1 DB ...

M

Max (@rozenmd)•2/6/25, 1:43 PM

We've seen a few reports of this over the past few days, what's your databaseId so we can investigate?

edit: nvm found it

Cct_roy Hi folks, i'm seeing 25-30 sec response times on basic inserts into a simple D1 ...

M

Max (@rozenmd)•2/6/25, 1:43 PM

We've seen a few reports of this over the past few days, what's your databaseId so we can investigate?

M

Max (@rozenmd)•2/6/25, 1:50 PM

For future reference:

M

Max (@rozenmd)•2/6/25, 1:50 PM

?databaseid

S

SuperHelpflare•2/6/25, 1:50 PM

Please post your D1 Database ID so we can help you debug your issue.

MMax (@rozenmd)just to help us look into this, is it also slow from the worker?

Z

zwily•2/6/25, 2:05 PM

It’s not slow anymore, but I’m a bit freaked out by that happening. Was going to go to production in a couple months

Zzwily It’s not slow anymore, but I’m a bit freaked out by that happening. Was going to...

M

Max (@rozenmd)•2/6/25, 2:05 PM

what's your databaseId?

MMax (@rozenmd)what's your databaseId?

Z

zwily•2/6/25, 2:06 PM

4a773374-363b-48c6-8044-df268ac46283

FFlikTeoh Would the DB in DO be considered more reliable in this case or it is going to en...

L

lambrospetrouOP•2/6/25, 3:21 PM

Same reliability exactly. Excluding human errors/bugs.

FFlikTeoh 😪 Is there a plan to make this better though?

M

Max (@rozenmd)•2/6/25, 3:38 PM

Yes.

FFlikTeoh Is there a time frame on this? Sorry for asking 🫣 been watching this for like 3...

L

lambrospetrouOP•2/6/25, 3:42 PM

Reliability doesn't have timelines. We constantly try to improve things as we find opportunities. There is an ongoing workstream for the forecoming months. There is nothing someone can promise you "by end of X it will be awesome".
If what we provide now suits your use-case, awesome

FFlikTeoh Totally understand, but D1 has been really unstable, at least for the recent 6-7...

L

lambrospetrouOP•2/6/25, 3:47 PM

Anything we promise won't fix it all of a sudden. Things will improve over time.

D1 has been really unstable

That's quite subjective though. If it's already that bad for your case, then maybe it's not the right product for that.

M

meow•2/6/25, 5:12 PM

Does anyone know the minimum set of permissions I need to give my API key for listing and creating new Cloudflare D1's? I've tried doing only read and edit for D1 and that doesn't seem to be enough.

Mmeow Does anyone know the minimum set of permissions I need to give my API key for li...

M

Max (@rozenmd)•2/6/25, 5:47 PM

Can you show a screenshot of the error you're seeing? I've just created a D1:edit API token in the dash and was able to create a D1 DB through the API

MMax (@rozenmd)We've seen a few reports of this over the past few days, what's your databaseId ...

K

kingmesal•2/6/25, 5:52 PM

We just experienced a rash of errors, not really random

We have a job that produces messages onto a queue. The queue is setup for batches of 10 events, auto concurrency, etc.. default settings

Our worker that picks up these events for this specific job may run as many as 20 total queries, well under the limits.

We made D1 (

f34b99be-6f7a-488b-a6a2-e449cc017d5e

f34b99be-6f7a-488b-a6a2-e449cc017d5e

f34b99be-6f7a-488b-a6a2-e449cc017d5e

f34b99be-6f7a-488b-a6a2-e449cc017d5e) explode!

Error: D1_ERROR: D1 DB is overloaded. Too many requests queued.

Error: D1_ERROR: D1 DB is overloaded. Too many requests queued.

Error: D1_ERROR: D1 DB is overloaded. Too many requests queued.

Error: D1_ERROR: D1 DB is overloaded. Too many requests queued.

About an hour ago, thousands and thousands of these errors were generated. In case it is helpful:

$metadata: {
  requestId: "F565DKCXDMGXX826",
  id: "01JKE3MTJY6KJKCZYX985BEH86",
  type: "cf-worker"
}

$metadata: {
  requestId: "F565DKCXDMGXX826",
  id: "01JKE3MTJY6KJKCZYX985BEH86",
  type: "cf-worker"
}

$metadata: {
  requestId: "F565DKCXDMGXX826",
  id: "01JKE3MTJY6KJKCZYX985BEH86",
  type: "cf-worker"
}

$metadata: {
  requestId: "F565DKCXDMGXX826",
  id: "01JKE3MTJY6KJKCZYX985BEH86",
  type: "cf-worker"
}

Kkingmesal We just experienced a rash of errors, not really random We have a job that pro...

M

Max (@rozenmd)•2/6/25, 6:02 PM

is the queue/worker still active?

Kkingmesal We just experienced a rash of errors, not really random We have a job that pro...

M

Max (@rozenmd)•2/6/25, 6:09 PM

It looks like you more than 100xed the number of queries per second your DB gets at 16:40 UTC, seems like you're actually overloading the system

MMax (@rozenmd)Can you show a screenshot of the error you're seeing? I've just created a D1:edi...

M

meow•2/6/25, 6:24 PM

I'm using the Cloudflare JS SDK in a project I just setup. The response I'm getting is

error: 400 {"success":false,"errors":[{"code":10001,"message":"Unable to authenticate request"}]}

error: 400 {"success":false,"errors":[{"code":10001,"message":"Unable to authenticate request"}]}

error: 400 {"success":false,"errors":[{"code":10001,"message":"Unable to authenticate request"}]}

error: 400 {"success":false,"errors":[{"code":10001,"message":"Unable to authenticate request"}]}. When I switch to my global key it works, but when I switch to my read & edit D1 key I get this auth error.

M

meow•2/6/25, 6:27 PM

The route is just

async (c) => {
    const cloudflare = c.get("cloudflare")
    const d1 = await cloudflare.d1.database.list(
      { account_id: "__ACCT_ID_HERE__" }
    )
    return c.json(d1)
  }

async (c) => {
    const cloudflare = c.get("cloudflare")
    const d1 = await cloudflare.d1.database.list(
      { account_id: "__ACCT_ID_HERE__" }
    )
    return c.json(d1)
  }

async (c) => {
    const cloudflare = c.get("cloudflare")
    const d1 = await cloudflare.d1.database.list(
      { account_id: "__ACCT_ID_HERE__" }
    )
    return c.json(d1)
  }

async (c) => {
    const cloudflare = c.get("cloudflare")
    const d1 = await cloudflare.d1.database.list(
      { account_id: "__ACCT_ID_HERE__" }
    )
    return c.json(d1)
  }

Mmeow I'm using the Cloudflare JS SDK in a project I just setup. The response I'm gett...

M

Max (@rozenmd)•2/6/25, 6:30 PM

Can you try make a token with just edit?

MMax (@rozenmd)It looks like you more than 100xed the number of queries per second your DB gets...

K

kingmesal•2/6/25, 6:34 PM

given that we have default settings for the queue (auto scale), is this an issue that say a few thousand messages all parallelized nearly simultaneously?