Cloudflare Developers•11mo ago

My frustration in here https://discord.com/channels/595317990191398933/992060581832032316/1332328652

My frustration in here
d1-database

戴戴きますReina My frustration in here https://discord.com/channels/595317990191398933/992060581...

森

森@VerseLink (RW)•1/31/25, 5:17 PM

I use drizzle for migration, so far no problems in my test environment

戴戴きますReina I already tried using prisma and drizzle for d1 and difficult to migrate, we can...

James•1/31/25, 5:47 PM

drizzle migrations works great for me with D1 and DO sqlite, i dont even use drizzle for the ORM, just the migrations and schema definition

JJames drizzle migrations works great for me with D1 and DO sqlite, i dont even use dri...

森

森@VerseLink (RW)•1/31/25, 5:52 PM

Same, I use Kysely for building complex query and raw SQL if it’s very simple operations

森

森@VerseLink (RW)•1/31/25, 5:52 PM

But drizzle migration helps alleviate a lot of headaches when maintaining SQL migrations

森森@VerseLink (RW)But drizzle migration helps alleviate a lot of headaches when maintaining SQL mi...

James•1/31/25, 5:54 PM

its also helpful, I re-use the schema definitions for type casting raw sql response data as well

verdverm.com•1/31/25, 10:59 PM

Is there an example that shows how one would talk to and execute SQL from a worker against a D1 when you use the recommendation to have something like a D1 per customer when using workers-for-platforms?

i40west•2/1/25, 2:12 AM

Is there a way to tell if an error thrown by D1 is due to a constraint violation? More generally, is there a way to detect a unique constraint violation? Everything seems to just be a D1_ERROR

Ii40west Is there a way to tell if an error thrown by D1 is due to a constraint violation...

森

森@VerseLink (RW)•2/1/25, 2:53 AM

I believe the error message would include SQLITE_CONSTRAINT?

森

森@VerseLink (RW)•2/1/25, 2:54 AM

https://www.sqlite.org/rescode.html#constraint

i40west•2/1/25, 3:00 AM

parsing the message isn't great, but it'll do

Khafra•2/2/25, 2:33 AM

Hi, I'm using triggers and encounter an "internal error" that times out after 30 seconds when the abort occurs. It's not a massive problem, but it would be nice if this was fixed.

The trigger:

CREATE TRIGGER deduct_coins_on_consumable
BEFORE INSERT ON ActiveConsumables
  BEGIN
    UPDATE accounts
    SET pseudoCoins = pseudoCoins - (SELECT cost FROM Consumables WHERE id = NEW.consumableId)
    WHERE
      id = NEW.accountId 
        AND
      pseudoCoins >= (SELECT cost FROM Consumables WHERE id = NEW.consumableId);

  SELECT RAISE(ROLLBACK, 'Insufficient coins') WHERE changes() = 0;
END

CREATE TRIGGER deduct_coins_on_consumable
BEFORE INSERT ON ActiveConsumables
  BEGIN
    UPDATE accounts
    SET pseudoCoins = pseudoCoins - (SELECT cost FROM Consumables WHERE id = NEW.consumableId)
    WHERE
      id = NEW.accountId 
        AND
      pseudoCoins >= (SELECT cost FROM Consumables WHERE id = NEW.consumableId);

  SELECT RAISE(ROLLBACK, 'Insufficient coins') WHERE changes() = 0;
END

the internal error occurs when coins < cost.

"database_name": "discord-oauth",
"database_id": "2bd61136-7eb6-4b5d-82ce-f3405294398d"

Khafra•2/2/25, 2:34 AM

afaict the trigger works fine, other than it timing out/taking 30 seconds to throw

KKhafra Hi, I'm using triggers and encounter an "internal error" that times out after 30...

Isaac McFadyen•2/2/25, 3:41 AM

D1 has no manual transactions so I expect the error is a result of you trying to manually roll back (unsure though)

Khafra•2/2/25, 4:11 AM

Right.. I'll see if RAISE(ABORT, ...) has the same issues.

Khafra•2/2/25, 4:12 AM

I think maybe there is still something to fix with the internal error I was getting though

ikiryo•2/2/25, 3:19 PM

Hi, I have followed the tutorial here - https://www.prisma.io/docs/orm/overview/databases/cloudflare-d1#3-generate-sql-statements-using-prisma-migrate-diff

I ran this command:

npx prisma migrate diff \
--from-empty \
--to-schema-datamodel ./prisma/schema.prisma \
--script \
--output migrations/0001_create_user_table.sql
It works great. However I don't want to apply this migration yet. I want to make some more changes to the schema (add a "color" column to the "User" model), and then I want to run npx prisma migrate diff to a 2nd file called migrations/0002_add_user_color.sql.

However the docs only show show how to do this with --from-local-d1. Is there a way to make npx prisma migrate diff just look at the existing migrations and go from there?

Cloudflare D1 | Prisma Documentation

Guide to Cloudflare D1

kamama•2/2/25, 3:20 PM

Any roadmap to support replication like Turso for D1?

Kkamama Any roadmap to support replication like Turso for D1?

lambrospetrou•2/2/25, 5:55 PM

d1-database

Original message was deleted

lambrospetrou•2/3/25, 11:05 AM

Focusing just on the Worker code, since the first is mostly Drizzle stuff I guess which I am not familiar. What is the

createDB()

createDB()

function?
Your log clearly shows that your D1 database is correctly bound to the worker. So, it's either the way Drizzle is used, or that

createDB

createDB

function messing up something.

Llambrospetrou Focusing just on the Worker code, since the first is mostly Drizzle stuff I gues...

Althaf•2/3/25, 11:23 AM

Hey thankyou for the response. I have finally fixed it

Llambrospetrou Focusing just on the Worker code, since the first is mostly Drizzle stuff I gues...

Althaf•2/3/25, 11:24 AM

It was bind name mismatch

kamama•2/3/25, 1:15 PM

Btw, anyway to create and bind D1 on runtime ?

Kkamama Btw, anyway to create and bind D1 on runtime ?

Max (@rozenmd)•2/3/25, 3:42 PM

only via the HTTP API

jd•2/3/25, 8:02 PM

hi qq -- im trying to convert a largely kv-based lookup metadata of ~7GB in size of json and gets 10+ GB in size later with parallel reads, updates. Metadata needs persistence storage for consistency and availability too.

My metadata has 10M+ entries where each entry is a small dict of mostly nested strings of a loose schema, mixed with floats.

Is this size a good use case for cloudflare d1 / kv? thanks!

Jjd hi qq -- im trying to convert a largely kv-based lookup metadata of ~7GB in size...

Max (@rozenmd)•2/3/25, 8:05 PM

I'd probably pick KV for that, but there's a doc to help you choose: https://developers.cloudflare.com/workers/platform/storage-options/#choose-a-storage-product

Cloudflare Docs

Choosing a data or storage product. · Cloudflare Workers docs

Storage and database options available on Cloudflare's developer platform.

sahilmob•2/4/25, 7:53 PM

Hi, I'm building a backend with hono and better-auth, since better auth needs a DB I can only pass the DB instance using I middleware so I wonder if it is ok I initialize better auth only once in cloudflare worker, I mean to create a singleton that lives as long as the worker do. here is an example:

export class Auth {
  static initialized = false;
  private static instance: ReturnType<typeof betterAuth>;

  static async init(c: Context<Env>) {
    if (Auth.instance) {
      await Auth.getSession(c);
      Auth.initialized = true;
      return Object.freeze(Auth.instance);
    }

    Auth.instance = betterAuth({
      database: drizzleAdapter(
        {
          dialect: new D1Dialect({
            database: c.env.DB,
          }),
        },
      ),
    });

    Auth.initialized = true;

    await Auth.getSession(c);

    return Object.freeze(Auth.instance);
  }

  static async getSession(c: Context) {
    const session = await Auth.instance.api.getSession({
      headers: c.req.raw.headers,
    });

    c.set("user", session?.user ?? null);
    c.set("session", session?.session ?? null);

    return session;
  }

  static getInstance() {
    if (!Auth.instance) {
      throw new Error("Auth instance not initialized");
    }

    return Object.freeze(Auth.instance);
  }
}

export class Auth {
  static initialized = false;
  private static instance: ReturnType<typeof betterAuth>;

  static async init(c: Context<Env>) {
    if (Auth.instance) {
      await Auth.getSession(c);
      Auth.initialized = true;
      return Object.freeze(Auth.instance);
    }

    Auth.instance = betterAuth({
      database: drizzleAdapter(
        {
          dialect: new D1Dialect({
            database: c.env.DB,
          }),
        },
      ),
    });

    Auth.initialized = true;

    await Auth.getSession(c);

    return Object.freeze(Auth.instance);
  }

  static async getSession(c: Context) {
    const session = await Auth.instance.api.getSession({
      headers: c.req.raw.headers,
    });

    c.set("user", session?.user ?? null);
    c.set("session", session?.session ?? null);

    return session;
  }

  static getInstance() {
    if (!Auth.instance) {
      throw new Error("Auth instance not initialized");
    }

    return Object.freeze(Auth.instance);
  }
}

and in the worker handler:

const app = new Hono<Env>();

app.use("*", async (c, next) => {
  if (Auth.initialized) return next();

  await Auth.init(c);
  return next();
});

app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw));

export default app;
export type ApiRoutes = typeof apiRoutes;

const app = new Hono<Env>();

app.use("*", async (c, next) => {
  if (Auth.initialized) return next();

  await Auth.init(c);
  return next();
});

app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw));

export default app;
export type ApiRoutes = typeof apiRoutes;

The rational behind this approach is to not reinitialize the auth instance and make a db connection for each request

Ssahilmob Hi, I'm building a backend with hono and better-auth, since better auth needs a ...

lambrospetrou•2/5/25, 11:02 AM

It's not recommended in general to reuse

env

env

across Worker requests, which is why you receive it in every request.
Also, D1 doesn't have the traditional TCP connection establishing as other databases like Postgres. Every D1 query you make uses our internal networking to reach your remote database anyway, there isn't a D1 connection concept at all, it's similar as calling a service binding/WorkerEntrypoint.

So, my suggestion is to not store env globally and reuse it, otherwise you might be seeing weird behavior occassionally which will be hard to debug.

ct_roy•2/5/25, 12:34 PM

Hi folks, i'm seeing 25-30 sec response times on basic inserts into a simple D1 table today. Is there something up with D1 at the moment? i'm also seeing frequent "Error: D1_ERROR: Network connection lost." messages.

kake•2/5/25, 4:58 PM

Hey guys, this is my first time using d1 and im getting > 1000ms latencies for simple selects. is this to be expected? ive seen other people having the same issue on the forum aswell but there seems to be no responses. even if i was on the other side of the world to the db the latency should be max 300ms. 1500ms is unacceptable for users in most cases. I have tried to access it from workers aswell with more success (getting down to 800ms) but this is still way too high

Khafra•2/5/25, 5:14 PM

Started getting this error and the database is extremely slow:

D1_ERROR: D1 DB is overloaded. Requests queued for too long., stack: Error: D1_ERROR: D1 DB is overloaded. Requests queued for too long.

D1_ERROR: D1 DB is overloaded. Requests queued for too long., stack: Error: D1_ERROR: D1 DB is overloaded. Requests queued for too long.

the error is happening on a batch query. Every column has an index. This started happening out of the blue, but it's not the first time I've had similar issues with d1 (the same database, different tables): workers-and-pages-helpD1 in workers constant query errors & timeouts

Kkake Hey guys, this is my first time using d1 and im getting > 1000ms latencies for s...

lambrospetrou•2/5/25, 5:17 PM

ive seen other people having the same issue on the forum aswell but there seems to be no responses

It's mentioned many times in the group actually why that is.

The REST API is not how you are meant to be using D1 at the moment as part of your production applications. To get the best performance, you should use the Workers binding and access D1 through your worker. The REST API goes through a centralized location and has different layers adding latency, more aggressive rate limiting, and is therefore not suitable to be used in your actual application to access D1 in a hot path.

There has been recent improvements in the D1 latencies when using the Worker binding and you get pretty much a roundtrip's worth of latency: https://developers.cloudflare.com/d1/platform/changelog/#2025-01-07

Cloudflare Docs

Changelog · Cloudflare D1 docs

Subscribe to RSS

Llambrospetrou > ive seen other people having the same issue on the forum aswell but there seem...

kake•2/5/25, 5:19 PM

i see, thank you for your reply

KKhafra Started getting this error and the database is extremely slow: `D1_ERROR: D1 DB ...

Khafra•2/5/25, 5:22 PM

bumping so it doesn't get hidden

krofdrakula•2/5/25, 7:52 PM

we've just had a massive hit on performance when adding a few new indexes on one of the tables in our D1 database because the query planner used the incorrect index. we pushed out a quick fix by forcing queries to use the old index using

INDEXED BY

INDEXED BY

while i read through the sqlite docs to determine what the correct course of action is.

reading through the query planner docs, i found the chapter on

ANALYZE

ANALYZE

and am wondering whether D1 periodically runs

PRAGMA optimize

PRAGMA optimize

. if not, is this something that should be done by us (the users of D1) periodically or after running migrations that would update tables and add/remove indexes?

zwily•2/6/25, 4:59 AM

is the d1 api struggling right now?

zwily•2/6/25, 5:02 AM

my D1 database has gotten super slow (bound to a worker) too... ugh

Zzwily is the d1 api struggling right now?

Max (@rozenmd)•2/6/25, 6:23 AM

just to help us look into this, is it also slow from the worker?

yinxingmaiming6409•2/6/25, 8:28 AM

Is there a problem with R2? Failed to upload images and access.

Nlea•2/6/25, 10:28 AM

Hello everyone!

@boots and I were discussing taking D1 with Drizzle to production during our stream yesterday. We are working on a setup and scripts to make the process as easy and seamless as possible. We would love to hear your feedback on our ideas :
https://www.youtube.com/watch?v=4x-AvV5x1mE&t=6s

YouTubeFiberplane

Honc Out #4: Taking D1 Into Production

In this episode of Honc Out, we’re taking D1 into production!

HONC comes with a D1 template that makes it easy to spin up D1 locally for development. However, when deploying a Cloudflare Worker to production, we need to create a D1 instance in Cloudflare and connect it to our deployed worker.

We’ll walk through the configuration steps to ...

Tthomfre I'm getting lots of D1 errors today, in addition to it being very slow. Random e...

Max (@rozenmd)•2/6/25, 1:42 PM

We've seen a few reports of this over the past few days, what's your databaseId so we can investigate?

KKhafra Started getting this error and the database is extremely slow: `D1_ERROR: D1 DB ...

Max (@rozenmd)•2/6/25, 1:43 PM

We've seen a few reports of this over the past few days, what's your databaseId so we can investigate?

edit: nvm found it

Cct_roy Hi folks, i'm seeing 25-30 sec response times on basic inserts into a simple D1 ...

Max (@rozenmd)•2/6/25, 1:43 PM

We've seen a few reports of this over the past few days, what's your databaseId so we can investigate?

Max (@rozenmd)•2/6/25, 1:50 PM

For future reference:

Max (@rozenmd)•2/6/25, 1:50 PM

?databaseid

SuperHelpflare•2/6/25, 1:50 PM

Please post your D1 Database ID so we can help you debug your issue.

MMax (@rozenmd)just to help us look into this, is it also slow from the worker?

zwily•2/6/25, 2:05 PM

It’s not slow anymore, but I’m a bit freaked out by that happening. Was going to go to production in a couple months

Zzwily It’s not slow anymore, but I’m a bit freaked out by that happening. Was going to...

Max (@rozenmd)•2/6/25, 2:05 PM

what's your databaseId?

CREATE TRIGGER deduct_coins_on_consumable BEFORE INSERT ON ActiveConsumables BEGIN UPDATE accounts SET pseudoCoins = pseudoCoins - (SELECT cost FROM Consumables WHERE id = NEW.consumableId) WHERE id = NEW.accountId AND pseudoCoins >= (SELECT cost FROM Consumables WHERE id = NEW.consumableId); SELECT RAISE(ROLLBACK, 'Insufficient coins') WHERE changes() = 0; END

export class Auth { static initialized = false; private static instance: ReturnType<typeof betterAuth>; static async init(c: Context<Env>) { if (Auth.instance) { await Auth.getSession(c); Auth.initialized = true; return Object.freeze(Auth.instance); } Auth.instance = betterAuth({ database: drizzleAdapter( { dialect: new D1Dialect({ database: c.env.DB, }), }, ), }); Auth.initialized = true; await Auth.getSession(c); return Object.freeze(Auth.instance); } static async getSession(c: Context) { const session = await Auth.instance.api.getSession({ headers: c.req.raw.headers, }); c.set("user", session?.user ?? null); c.set("session", session?.session ?? null); return session; } static getInstance() { if (!Auth.instance) { throw new Error("Auth instance not initialized"); } return Object.freeze(Auth.instance); } }

const app = new Hono<Env>(); app.use("*", async (c, next) => { if (Auth.initialized) return next(); await Auth.init(c); return next(); }); app.all("/api/**", (c) => Auth.getInstance().handler(c.req.raw)); export default app; export type ApiRoutes = typeof apiRoutes;

My frustration in here https://discord.com/channels/595317990191398933/992060581832032316/1332328652

Similar Threads