Cloudflare Developers•9mo ago

@jacky @hoptv @언블베 is this still happening?

MMilan @jacky @hoptv @언블베 is this still happening?

언

언블베•4/23/25, 4:34 PM

Its still happening

언언블베 Its still happening

lambrospetrou•4/23/25, 4:38 PM

Is it happening continuously 100% of the time or periodically a couple of requests failing?
Also, what is happening exactly? The

network connection lost

network connection lost

errors, or are you getting

internal error

internal error

errors? Those are two very different things.

jacky•4/23/25, 6:02 PM

in my situation it was completely offline for like 30 min, during that time we are getting both of these errors and these are not happening in the same request

jacky•4/23/25, 6:03 PM

and can confirm this was a account specific issue, my blog is running well on my personal cf account, my company's account, not that lucky...

jacky•4/23/25, 6:24 PM

i see that there was an ssl certificate provisioning delay issue posted on status page at that moment, could that be the cause?

Jjacky i see that there was an ssl certificate provisioning delay issue posted on statu...

Chaika•4/23/25, 6:35 PM

No, wouldn't do it
Providing DB IDs could be helpful for Cf Employees. Not sure if there's enough continued reports to escalate separately.
Did you have Read Replicas on? They still have a way higher amount of transient errors as far as I see, although better then last week by a lot

Kavatch•4/23/25, 6:42 PM

To whomever from Cloudflare can fix this typo:
In German the information about how many lines can be written per month says it

50 Milliarden

50 Milliarden

which would translate to

50 Billion

50 Billion

. Although then English version says

50 Million

50 Million

like pretty much all other resources

CChaika No, wouldn't do it Providing DB IDs could be helpful for Cf Employees. Not sure ...

jacky•4/24/25, 3:32 AM

no we didn't have read replica enabled and won't enable until GA, the db id is : 1165e434-3870-464d-89d4-6c3be0f0a64f

juanferreras•4/24/25, 4:25 PM

is this type of singleton to not re-initialize connections on every requests disencouraged in Workers? Unless I change it to be a map by bookmark I'm assuming implementing Sessions API implicitly means I'll be always connecting on each request (and/or potentially some internal optimizations makes this unnecessary)

let db: Kysely<DB>;
export async function getDB() {
  if (db) return db;
  const d1Binding = await getDatabaseProvider();
  db = new Kysely<DB>({
    dialect: new D1Dialect({ database: d1Binding }),
  });
  return db;
}

let db: Kysely<DB>;
export async function getDB() {
  if (db) return db;
  const d1Binding = await getDatabaseProvider();
  db = new Kysely<DB>({
    dialect: new D1Dialect({ database: d1Binding }),
  });
  return db;
}

juanferreras•4/24/25, 4:29 PM

I guess workers can directly RPC to D1 so my old singleton is probably adding no benefit

Jjuanferreras is this type of singleton to not re-initialize connections on every requests dis...

lambrospetrou•4/24/25, 4:36 PM

I don't know Kysely, but your D1 queries are always going through the same process of sending a remote request regardless of what you do on your side. It's not like the traditional databases where you keepalive connections yourself.
If creating that Kysely object is expensive, maybe cache that, otherwise you don't get any benefit D1-wise.

Llambrospetrou I don't know Kysely, but your D1 queries are always going through the same proce...

juanferreras•4/24/25, 4:38 PM

excellent, thanks for the response!

aza547•4/24/25, 8:13 PM

I'm seeing lots of worker errors with

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

and

Error: D1_ERROR: Network connection lost.

Error: D1_ERROR: Network connection lost.

. I suspect the latter is a symptom of the former. Am I working my DB too hard? DB ID is

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

if that's any use

Aaza547 I'm seeing lots of worker errors with `Error: D1_ERROR: D1 DB's isolate exceeded...

lambrospetrou•4/24/25, 8:19 PM

What queries are you doing? If you are buffering or returning lots of MBs memory becomes an issue quickly.

aza547•4/24/25, 8:20 PM

I am basically just dumping an entire table out to json which I think will be the worst offender

aza547•4/24/25, 8:20 PM

Probably several MB at a time per request

aza547•4/24/25, 8:24 PM

What is the process memory limit for a D1 DB?

aza547•4/24/25, 8:24 PM

I can't see that in the docs but might be missing the obvious

aza547•4/24/25, 9:05 PM

Gotcha, probably a symptom of my service growing a bit and it all being on 1 D1 instance currently for no reason other than I thought I had more time to deal with that (was aware of the 10GB storage limit and currently at 800MB, but I guess this is the first bottleneck and I should start using multiple instances)

Aaza547 Gotcha, probably a symptom of my service growing a bit and it all being on 1 D1 ...

gruntlord6•4/24/25, 11:24 PM

You could also copy that data to another table first

gruntlord6•4/24/25, 11:24 PM

Sorry database

aza547•4/25/25, 9:38 AM

That's my plan yeah

aza547•4/25/25, 9:39 AM

Although enabling read replicas could maybe solve this too, but I've been scared off by the experimental warnings in the dashboard

KKavatch To whomever from Cloudflare can fix this typo: In German the information about ...

Kavatch•4/25/25, 12:05 PM

@lambrospetrou FYI (From your description, it seemed like you are probably the best person to know about this)

KKavatch Empty message

lambrospetrou•4/25/25, 12:11 PM

Is that in the D1 overview page?

Llambrospetrou Is that in the D1 overview page?

Kavatch•4/25/25, 12:17 PM

Yes correct

DDiego Hey all, just a quick question. I was able to create and populate a FTS5 virtual...

lambrospetrou•4/25/25, 3:30 PM

Every D1 response contains a

meta

meta

object that has this information about rows read and written and query duration itself.

lambrospetrou•4/25/25, 3:57 PM

Hey folks, the main fix for the increased

internal error

internal error

errors due to D1 read replication is now fully rolled out in production.

Feel free to re-enable replication again, and give us feedback.

Same guideline applies, D1 Read replication is in BETA. Do not enable directly in production without testing in a staging/test environment for a few days to ensure it works fine for your workload.

Read https://developers.cloudflare.com/d1/best-practices/read-replication/ for how to use.

Aajgeiss0702 thank you. Will there be any kind of notification when these fixes are rolled ou...

MilanOP•4/25/25, 4:03 PM

See Lambros' latest message.

ac•4/25/25, 7:10 PM

How can I copy a D1 database to another D1 database? I've tried using

wrangler d1 export

wrangler d1 export

to create a .sql file and then

wrangler d1 execute

wrangler d1 execute

with that file, but that fails every time due to foreign key constraints.

I know how to do it locally by just running sqlite3 on the sql file and replacing the file, but for remote changes it's confusing why the d1 exports generate invalid SQL for the

execute

execute

command. Any tips?

ac•4/25/25, 7:12 PM

Before anyone mentions it, yeah the export includes

PRAGMA defer_foreign_keys=TRUE;

PRAGMA defer_foreign_keys=TRUE;

at the top

ac•4/25/25, 7:24 PM

Looks like it works if I change that to PRAGMA foreign_keys = off;PRAGMA foreign_keys = off; and turn off FKs completely for the duration of the file (then on again at the end)

ac•4/25/25, 8:04 PM

Ok, separate question: is there any way to programmatically know what local sqlite files match to which DB bindings for a worker? I have multiple that all persist to the same directory and the sqlite file names are all hashed and don't seem related to the database ID or anything

gruntlord6•4/25/25, 9:21 PM

why not just have a worker batch the contents?

seflanka•4/25/25, 10:27 PM

Is there any ETA on when Cloudflare SQL D1 clusters will be available in Brazil? When can we expect Cloudflare SQL D1 infrastructure to be available in Brazil?

Chaika•4/25/25, 10:48 PM

?crossposting -> workers-and-pages-discussions

SuperHelpflare•4/25/25, 10:48 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Llambrospetrou Hey folks, the main fix for the increased `internal error` errors due to D1 read...

Zack•4/26/25, 12:25 AM

I've just rolled up our production load over the last hour to 100% and so far, zero errors!

Zack•4/26/25, 12:26 AM

Initially had some increased latency but now presumably the database has "warmed up" and we are back to the same latency as on our pre-replica setup!

Zack•4/26/25, 12:27 AM

Going to keep an eye on it, but will probably leave it running for the next 12 hours and check how it looks in the morning from our monitoring

parisholley•4/26/25, 1:53 PM

would enabling read replicas allow us to do an export of the database without blocking requests?

Pparisholley would enabling read replicas allow us to do an export of the database without bl...

lambrospetrou•4/26/25, 2:46 PM

Not yet. We will be working on that soon.

Zack•4/26/25, 7:37 PM

Some more feedback: I reverted back to our original setup after a few hours, though we saw no errors, latencies did increase more than I would have liked. Note that these graph quantiles aren't accurate for their quantile due to the way the metrics are currently aggregated, but the difference between the before and after is accurate and does show a significant shift in our latencies

Zack•4/26/25, 7:37 PM

However we experienced zero errors whatsoever, so that is a significant improvement from before!

Zack•4/26/25, 7:38 PM

I think I asked this question before, but I don't think it was answered. For the replicas, when they startup, do they have to go back to the origin DC for all of the data they need, or just for any recent checkpoints they are missing? I ask this because I think the reason we are getting these latency spikes is for regions where it was currently nighttime and the requests were low

let db: Kysely<DB>; export async function getDB() { if (db) return db; const d1Binding = await getDatabaseProvider(); db = new Kysely<DB>({ dialect: new D1Dialect({ database: d1Binding }), }); return db; }

@jacky @hoptv @언블베 is this still happening?

Similar Threads