Just wondering if there is an update on the fixes that were discussed for d1 replicas. I believe the

ZZack Just wondering if there is an update on the fixes that were discussed for d1 rep...

L

lambrospetrou•4/23/25, 11:52 AM

A preliminary quick fix was rolled out indeed last week that would improve certain cases, and the main fix is currently gradually rolling out but will take a few days to make sure nothing else breaks.

Llambrospetrou A preliminary quick fix was rolled out indeed last week that would improve certa...

Z

ZackOP•4/23/25, 11:53 AM

Great to hear, would you be able to post here when that is fully rolled out? I would love to test replicas again after that with a 50/50 rollout in our prod environment

Llambrospetrou What is `new_databases` under the `[[migrations]]`? The `[[migrations]]` block i...

J

Jk3ndo•4/23/25, 12:01 PM

Yes sure, here is the class responsible of database operations
I have removed as well the D1 database from migrations section but it still provide the same issue.
@nora , here attached you can find my function where I use the prepare method (websiteVerificationReport.ts) and the entry point of the worker.

ZZack Great to hear, would you be able to post here when that is fully rolled out? I w...

L

lambrospetrou•4/23/25, 12:02 PM

I will post here, yes. Although, again, I do strongly suggest you try it on a staging/test environment first.

Llambrospetrou I will post here, yes. Although, again, I do strongly suggest you try it on a st...

Z

ZackOP•4/23/25, 12:02 PM

Yea I will do ofc

Z

ZackOP•4/23/25, 12:02 PM

However the problem only presented it to us once we hit it with significant load

Z

ZackOP•4/23/25, 12:03 PM

So will roll up from 5% -> 50% and leave it there for a while

JJk3ndo Yes sure, here is the class responsible of database operations I have removed as...

L

lambrospetrou•4/23/25, 12:07 PM

You don't show how

WebsiteVerificationReport.create(..)

WebsiteVerificationReport.create(..)

is called, which is the one causing you issues. The error implies that you are passing the wrong argument for

.
~~If that's meant to be Hono context, you need to get the env out of it, bindings are not directly in the context.~~ (edit: I see you do use it as Hono context)

J

Jk3ndo•4/23/25, 12:11 PM

@lambrospetrou Here is where I call it

export async function post(c: any, resultCheck?: IWebsiteVerificationReport, db?: any) {
    let data;
    try {
        if (resultCheck?.domain) {
            data = resultCheck;
        } else {
            data = await c.req.json();
        }
    } catch (error) {
        return c.text('Invalid JSON input', 400);
    }
    if (!data.domain || !data.slug || !data.verification || typeof data.status != 'boolean') {
        return new Response('Missing required fields', {status: 400});
        // return c.text('Missing required fields', 400);
    }
    try {
        const report = WebsiteVerificationReport.fromObject(data);
        const result = await WebsiteVerificationReport.create(report, c, db);
        return new Response(`${{ success: true, result }}`, {status: 201})
        // return c.json({ success: true, result }, 201);
    }
    catch (error) {
        console.error("Error:", error);
        return new Response('Error inserting data into database', {status: 500});
        // return c.text("Error inserting data into database", 500);
    }
}

export async function post(c: any, resultCheck?: IWebsiteVerificationReport, db?: any) {
    let data;
    try {
        if (resultCheck?.domain) {
            data = resultCheck;
        } else {
            data = await c.req.json();
        }
    } catch (error) {
        return c.text('Invalid JSON input', 400);
    }
    if (!data.domain || !data.slug || !data.verification || typeof data.status != 'boolean') {
        return new Response('Missing required fields', {status: 400});
        // return c.text('Missing required fields', 400);
    }
    try {
        const report = WebsiteVerificationReport.fromObject(data);
        const result = await WebsiteVerificationReport.create(report, c, db);
        return new Response(`${{ success: true, result }}`, {status: 201})
        // return c.json({ success: true, result }, 201);
    }
    catch (error) {
        console.error("Error:", error);
        return new Response('Error inserting data into database', {status: 500});
        // return c.text("Error inserting data into database", 500);
    }
}

export async function post(c: any, resultCheck?: IWebsiteVerificationReport, db?: any) {
    let data;
    try {
        if (resultCheck?.domain) {
            data = resultCheck;
        } else {
            data = await c.req.json();
        }
    } catch (error) {
        return c.text('Invalid JSON input', 400);
    }
    if (!data.domain || !data.slug || !data.verification || typeof data.status != 'boolean') {
        return new Response('Missing required fields', {status: 400});
        // return c.text('Missing required fields', 400);
    }
    try {
        const report = WebsiteVerificationReport.fromObject(data);
        const result = await WebsiteVerificationReport.create(report, c, db);
        return new Response(`${{ success: true, result }}`, {status: 201})
        // return c.json({ success: true, result }, 201);
    }
    catch (error) {
        console.error("Error:", error);
        return new Response('Error inserting data into database', {status: 500});
        // return c.text("Error inserting data into database", 500);
    }
}

export async function post(c: any, resultCheck?: IWebsiteVerificationReport, db?: any) {
    let data;
    try {
        if (resultCheck?.domain) {
            data = resultCheck;
        } else {
            data = await c.req.json();
        }
    } catch (error) {
        return c.text('Invalid JSON input', 400);
    }
    if (!data.domain || !data.slug || !data.verification || typeof data.status != 'boolean') {
        return new Response('Missing required fields', {status: 400});
        // return c.text('Missing required fields', 400);
    }
    try {
        const report = WebsiteVerificationReport.fromObject(data);
        const result = await WebsiteVerificationReport.create(report, c, db);
        return new Response(`${{ success: true, result }}`, {status: 201})
        // return c.json({ success: true, result }, 201);
    }
    catch (error) {
        console.error("Error:", error);
        return new Response('Error inserting data into database', {status: 500});
        // return c.text("Error inserting data into database", 500);
    }
}

J

Jk3ndo•4/23/25, 12:12 PM

const result = await WebsiteVerificationReport.create(report, c, db);

J

Jk3ndo•4/23/25, 12:15 PM

The post method itself it called by saveReport:

export const saveReport = async(context: any, data: any, domain: string, db?: any) => {
    try {
        console.log(JSON.stringify(`App in ctx is ${JSON.stringify(context)}`))
        await post(context!, data, db);
    } catch (error) {
        console.error(error);
        return new Response(`Error while managing ckecks for website ${domain}. \n Message: ${error}.`, {status: 500} );
    }
}

export const saveReport = async(context: any, data: any, domain: string, db?: any) => {
    try {
        console.log(JSON.stringify(`App in ctx is ${JSON.stringify(context)}`))
        await post(context!, data, db);
    } catch (error) {
        console.error(error);
        return new Response(`Error while managing ckecks for website ${domain}. \n Message: ${error}.`, {status: 500} );
    }
}

export const saveReport = async(context: any, data: any, domain: string, db?: any) => {
    try {
        console.log(JSON.stringify(`App in ctx is ${JSON.stringify(context)}`))
        await post(context!, data, db);
    } catch (error) {
        console.error(error);
        return new Response(`Error while managing ckecks for website ${domain}. \n Message: ${error}.`, {status: 500} );
    }
}

export const saveReport = async(context: any, data: any, domain: string, db?: any) => {
    try {
        console.log(JSON.stringify(`App in ctx is ${JSON.stringify(context)}`))
        await post(context!, data, db);
    } catch (error) {
        console.error(error);
        return new Response(`Error while managing ckecks for website ${domain}. \n Message: ${error}.`, {status: 500} );
    }
}

which is called by the durable object like this:

export class RequestQueue {
    private queue: Array<any> = [];
    private isProcessing = false;
  
    async fetch(request: Request): Promise<Response> {
      const { method } = request;
  
      if (method === 'POST') {
        const payload = await request.json<LightModeQueueObject>();
        console.log(`Received payload: ${JSON.stringify(payload)}`); // Log the payload for debugging

        // Validate the payload
        if (!payload || !payload.proxyUrl || !payload.data || !payload.bearerToken) {
          console.error('Invalid payload:', payload);
          return new Response('Invalid payload', { status: 400 });
        }

        this.queue.push(payload);
        this.processQueue();
        return new Response('Enqueued');
      }
  
      return new Response('Method not allowed', { status: 405 });
    }
  
    private async processQueue() {
      if (this.isProcessing || this.queue.length === 0) return;
  
      this.isProcessing = true;
      const payload = this.queue.shift();
  
      try {
        let data;

        let result: any = await checkLightMode(payload.proxyUrl, payload.data, payload.bearerToken);
        await saveReport(payload.context, data, payload.data.domain.replace('https://', '').replace('http://', ''));

      } catch (error) {
        console.error(`Error processing payload: ${JSON.stringify(payload)} - ${error}`);
      }
    }
  }

export class RequestQueue {
    private queue: Array<any> = [];
    private isProcessing = false;
  
    async fetch(request: Request): Promise<Response> {
      const { method } = request;
  
      if (method === 'POST') {
        const payload = await request.json<LightModeQueueObject>();
        console.log(`Received payload: ${JSON.stringify(payload)}`); // Log the payload for debugging

        // Validate the payload
        if (!payload || !payload.proxyUrl || !payload.data || !payload.bearerToken) {
          console.error('Invalid payload:', payload);
          return new Response('Invalid payload', { status: 400 });
        }

        this.queue.push(payload);
        this.processQueue();
        return new Response('Enqueued');
      }
  
      return new Response('Method not allowed', { status: 405 });
    }
  
    private async processQueue() {
      if (this.isProcessing || this.queue.length === 0) return;
  
      this.isProcessing = true;
      const payload = this.queue.shift();
  
      try {
        let data;

        let result: any = await checkLightMode(payload.proxyUrl, payload.data, payload.bearerToken);
        await saveReport(payload.context, data, payload.data.domain.replace('https://', '').replace('http://', ''));

      } catch (error) {
        console.error(`Error processing payload: ${JSON.stringify(payload)} - ${error}`);
      }
    }
  }

export class RequestQueue {
    private queue: Array<any> = [];
    private isProcessing = false;
  
    async fetch(request: Request): Promise<Response> {
      const { method } = request;
  
      if (method === 'POST') {
        const payload = await request.json<LightModeQueueObject>();
        console.log(`Received payload: ${JSON.stringify(payload)}`); // Log the payload for debugging

        // Validate the payload
        if (!payload || !payload.proxyUrl || !payload.data || !payload.bearerToken) {
          console.error('Invalid payload:', payload);
          return new Response('Invalid payload', { status: 400 });
        }

        this.queue.push(payload);
        this.processQueue();
        return new Response('Enqueued');
      }
  
      return new Response('Method not allowed', { status: 405 });
    }
  
    private async processQueue() {
      if (this.isProcessing || this.queue.length === 0) return;
  
      this.isProcessing = true;
      const payload = this.queue.shift();
  
      try {
        let data;

        let result: any = await checkLightMode(payload.proxyUrl, payload.data, payload.bearerToken);
        await saveReport(payload.context, data, payload.data.domain.replace('https://', '').replace('http://', ''));

      } catch (error) {
        console.error(`Error processing payload: ${JSON.stringify(payload)} - ${error}`);
      }
    }
  }

export class RequestQueue {
    private queue: Array<any> = [];
    private isProcessing = false;
  
    async fetch(request: Request): Promise<Response> {
      const { method } = request;
  
      if (method === 'POST') {
        const payload = await request.json<LightModeQueueObject>();
        console.log(`Received payload: ${JSON.stringify(payload)}`); // Log the payload for debugging

        // Validate the payload
        if (!payload || !payload.proxyUrl || !payload.data || !payload.bearerToken) {
          console.error('Invalid payload:', payload);
          return new Response('Invalid payload', { status: 400 });
        }

        this.queue.push(payload);
        this.processQueue();
        return new Response('Enqueued');
      }
  
      return new Response('Method not allowed', { status: 405 });
    }
  
    private async processQueue() {
      if (this.isProcessing || this.queue.length === 0) return;
  
      this.isProcessing = true;
      const payload = this.queue.shift();
  
      try {
        let data;

        let result: any = await checkLightMode(payload.proxyUrl, payload.data, payload.bearerToken);
        await saveReport(payload.context, data, payload.data.domain.replace('https://', '').replace('http://', ''));

      } catch (error) {
        console.error(`Error processing payload: ${JSON.stringify(payload)} - ${error}`);
      }
    }
  }

J

Jk3ndo•4/23/25, 12:17 PM

I call the getLightModeChecks like this:

export async function getLightModesChecks(c: Context<{ Bindings: Env }>)

export async function getLightModesChecks(c: Context<{ Bindings: Env }>)

And inside here is how I call my durable object:

const payload: LightModeQueueObject = {
                data: config,
                proxyUrl: c.env.PROXY_API_URL,
                bearerToken: bearerToken,
                document: config.document!,
                context: c,
                db: c.env.DB
            }

            await queue.fetch('https://dummy-url/queue', { // The url is just a placeholder, it won't be used.
                method: 'POST',
                body: JSON.stringify(payload),
            });

const payload: LightModeQueueObject = {
                data: config,
                proxyUrl: c.env.PROXY_API_URL,
                bearerToken: bearerToken,
                document: config.document!,
                context: c,
                db: c.env.DB
            }

            await queue.fetch('https://dummy-url/queue', { // The url is just a placeholder, it won't be used.
                method: 'POST',
                body: JSON.stringify(payload),
            });

const payload: LightModeQueueObject = {
                data: config,
                proxyUrl: c.env.PROXY_API_URL,
                bearerToken: bearerToken,
                document: config.document!,
                context: c,
                db: c.env.DB
            }

            await queue.fetch('https://dummy-url/queue', { // The url is just a placeholder, it won't be used.
                method: 'POST',
                body: JSON.stringify(payload),
            });

const payload: LightModeQueueObject = {
                data: config,
                proxyUrl: c.env.PROXY_API_URL,
                bearerToken: bearerToken,
                document: config.document!,
                context: c,
                db: c.env.DB
            }

            await queue.fetch('https://dummy-url/queue', { // The url is just a placeholder, it won't be used.
                method: 'POST',
                body: JSON.stringify(payload),
            });

J

jacky•4/23/25, 12:59 PM

hi, we are getting these kinda problem:

D1_ERROR: internal error; reference = 353paq1aanuq0lqv4msq1jqt

---

D1_ERROR: Network connection lost.

---

can anybody help?

Jjacky hi, we are getting these kinda problem: D1_ERROR: internal error; reference = 3...

J

jacky•4/23/25, 1:00 PM

it's pretty urgent!

Jjacky hi, we are getting these kinda problem: D1_ERROR: internal error; reference = 3...

H

hoptv•4/23/25, 1:05 PM

I'm also experiencing the same issue.

언

언블베•4/23/25, 2:08 PM

Me too

M

Milan•4/23/25, 4:23 PM

@jacky @hoptv @언블베 is this still happening?

MMilan @jacky @hoptv @언블베 is this still happening?

언

언블베•4/23/25, 4:34 PM

Its still happening

언언블베 Its still happening

L

lambrospetrou•4/23/25, 4:38 PM

Is it happening continuously 100% of the time or periodically a couple of requests failing?
Also, what is happening exactly? The

network connection lost

network connection lost

errors, or are you getting

internal error

internal error

errors? Those are two very different things.

J

jacky•4/23/25, 6:02 PM

in my situation it was completely offline for like 30 min, during that time we are getting both of these errors and these are not happening in the same request

J

jacky•4/23/25, 6:03 PM

and can confirm this was a account specific issue, my blog is running well on my personal cf account, my company's account, not that lucky...

J

jacky•4/23/25, 6:24 PM

i see that there was an ssl certificate provisioning delay issue posted on status page at that moment, could that be the cause?

Jjacky i see that there was an ssl certificate provisioning delay issue posted on statu...

C

Chaika•4/23/25, 6:35 PM

No, wouldn't do it
Providing DB IDs could be helpful for Cf Employees. Not sure if there's enough continued reports to escalate separately.
Did you have Read Replicas on? They still have a way higher amount of transient errors as far as I see, although better then last week by a lot

K

Kavatch•4/23/25, 6:42 PM

To whomever from Cloudflare can fix this typo:
In German the information about how many lines can be written per month says it

50 Milliarden

50 Milliarden

which would translate to

50 Billion

50 Billion

. Although then English version says

50 Million

50 Million

like pretty much all other resources

CChaika No, wouldn't do it Providing DB IDs could be helpful for Cf Employees. Not sure ...

J

jacky•4/24/25, 3:32 AM

no we didn't have read replica enabled and won't enable until GA, the db id is : 1165e434-3870-464d-89d4-6c3be0f0a64f

J

juanferreras•4/24/25, 4:25 PM

is this type of singleton to not re-initialize connections on every requests disencouraged in Workers? Unless I change it to be a map by bookmark I'm assuming implementing Sessions API implicitly means I'll be always connecting on each request (and/or potentially some internal optimizations makes this unnecessary)

let db: Kysely<DB>;
export async function getDB() {
  if (db) return db;
  const d1Binding = await getDatabaseProvider();
  db = new Kysely<DB>({
    dialect: new D1Dialect({ database: d1Binding }),
  });
  return db;
}

let db: Kysely<DB>;
export async function getDB() {
  if (db) return db;
  const d1Binding = await getDatabaseProvider();
  db = new Kysely<DB>({
    dialect: new D1Dialect({ database: d1Binding }),
  });
  return db;
}

let db: Kysely<DB>;
export async function getDB() {
  if (db) return db;
  const d1Binding = await getDatabaseProvider();
  db = new Kysely<DB>({
    dialect: new D1Dialect({ database: d1Binding }),
  });
  return db;
}

let db: Kysely<DB>;
export async function getDB() {
  if (db) return db;
  const d1Binding = await getDatabaseProvider();
  db = new Kysely<DB>({
    dialect: new D1Dialect({ database: d1Binding }),
  });
  return db;
}

J

juanferreras•4/24/25, 4:29 PM

I guess workers can directly RPC to D1 so my old singleton is probably adding no benefit

Jjuanferreras is this type of singleton to not re-initialize connections on every requests dis...

L

lambrospetrou•4/24/25, 4:36 PM

I don't know Kysely, but your D1 queries are always going through the same process of sending a remote request regardless of what you do on your side. It's not like the traditional databases where you keepalive connections yourself.
If creating that Kysely object is expensive, maybe cache that, otherwise you don't get any benefit D1-wise.

Llambrospetrou I don't know Kysely, but your D1 queries are always going through the same proce...

J

juanferreras•4/24/25, 4:38 PM

excellent, thanks for the response!

A

aza547•4/24/25, 8:13 PM

I'm seeing lots of worker errors with

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

and

Error: D1_ERROR: Network connection lost.

Error: D1_ERROR: Network connection lost.

. I suspect the latter is a symptom of the former. Am I working my DB too hard? DB ID is

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

if that's any use

Aaza547 I'm seeing lots of worker errors with `Error: D1_ERROR: D1 DB's isolate exceeded...

L

lambrospetrou•4/24/25, 8:19 PM

What queries are you doing? If you are buffering or returning lots of MBs memory becomes an issue quickly.

A

aza547•4/24/25, 8:20 PM

I am basically just dumping an entire table out to json which I think will be the worst offender

A

aza547•4/24/25, 8:20 PM

Probably several MB at a time per request

A

aza547•4/24/25, 8:24 PM

What is the process memory limit for a D1 DB?

A

aza547•4/24/25, 8:24 PM

I can't see that in the docs but might be missing the obvious

A

aza547•4/24/25, 9:05 PM

Gotcha, probably a symptom of my service growing a bit and it all being on 1 D1 instance currently for no reason other than I thought I had more time to deal with that (was aware of the 10GB storage limit and currently at 800MB, but I guess this is the first bottleneck and I should start using multiple instances)

Aaza547 Gotcha, probably a symptom of my service growing a bit and it all being on 1 D1 ...

G

gruntlord6•4/24/25, 11:24 PM

You could also copy that data to another table first

G

gruntlord6•4/24/25, 11:24 PM

Sorry database

A

aza547•4/25/25, 9:38 AM

That's my plan yeah

A

aza547•4/25/25, 9:39 AM

Although enabling read replicas could maybe solve this too, but I've been scared off by the experimental warnings in the dashboard

KKavatch To whomever from Cloudflare can fix this typo: In German the information about ...

K

Kavatch•4/25/25, 12:05 PM

K

Kavatch•4/25/25, 12:05 PM

@lambrospetrou FYI (From your description, it seemed like you are probably the best person to know about this)

KKavatch Empty message

L

lambrospetrou•4/25/25, 12:11 PM

Is that in the D1 overview page?

Llambrospetrou Is that in the D1 overview page?

K

Kavatch•4/25/25, 12:17 PM

Yes correct

DDiego Hey all, just a quick question. I was able to create and populate a FTS5 virtual...

L

lambrospetrou•4/25/25, 3:30 PM

Every D1 response contains a

meta

meta

object that has this information about rows read and written and query duration itself.

L

lambrospetrou•4/25/25, 3:57 PM

Hey folks, the main fix for the increased

internal error

internal error

errors due to D1 read replication is now fully rolled out in production.

Feel free to re-enable replication again, and give us feedback.

Same guideline applies, D1 Read replication is in BETA. Do not enable directly in production without testing in a staging/test environment for a few days to ensure it works fine for your workload.

Read https://developers.cloudflare.com/d1/best-practices/read-replication/ for how to use.

Aajgeiss0702 thank you. Will there be any kind of notification when these fixes are rolled ou...

M

Milan•4/25/25, 4:03 PM

See Lambros' latest message.

Just wondering if there is an update on the fixes that were discussed for d1 replicas. I believe the

Similar Threads