I'm seeing lots of worker errors with `Error: D1_ERROR: D1 DB's isolate exceeded its memory limit an

I'm seeing lots of worker errors with

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset.

Error: D1_ERROR: D1 DB's isolate exceeded its memory limit and was reset. and

Error: D1_ERROR: Network connection lost.

Error: D1_ERROR: Network connection lost.

Error: D1_ERROR: Network connection lost.

Error: D1_ERROR: Network connection lost. . I suspect the latter is a symptom of the former. Am I working my DB too hard? DB ID is

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

1af9dfd0-4a92-4ed3-a17d-f3b09713d888

1af9dfd0-4a92-4ed3-a17d-f3b09713d888 if that's any use

Aaza547 I'm seeing lots of worker errors with `Error: D1_ERROR: D1 DB's isolate exceeded...

lambrospetrou•4/24/25, 8:19 PM

What queries are you doing? If you are buffering or returning lots of MBs memory becomes an issue quickly.

aza547OP•4/24/25, 8:20 PM

I am basically just dumping an entire table out to json which I think will be the worst offender

aza547OP•4/24/25, 8:20 PM

Probably several MB at a time per request

aza547OP•4/24/25, 8:24 PM

What is the process memory limit for a D1 DB?

aza547OP•4/24/25, 8:24 PM

I can't see that in the docs but might be missing the obvious

aza547OP•4/24/25, 9:05 PM

Gotcha, probably a symptom of my service growing a bit and it all being on 1 D1 instance currently for no reason other than I thought I had more time to deal with that (was aware of the 10GB storage limit and currently at 800MB, but I guess this is the first bottleneck and I should start using multiple instances)

Aaza547 Gotcha, probably a symptom of my service growing a bit and it all being on 1 D1 ...

gruntlord6•4/24/25, 11:24 PM

You could also copy that data to another table first

gruntlord6•4/24/25, 11:24 PM

Sorry database

aza547OP•4/25/25, 9:38 AM

That's my plan yeah

aza547OP•4/25/25, 9:39 AM

Although enabling read replicas could maybe solve this too, but I've been scared off by the experimental warnings in the dashboard

KKavatch To whomever from Cloudflare can fix this typo: In German the information about ...

Kavatch•4/25/25, 12:05 PM

@lambrospetrou FYI (From your description, it seemed like you are probably the best person to know about this)

KKavatch Empty message

lambrospetrou•4/25/25, 12:11 PM

Is that in the D1 overview page?

Llambrospetrou Is that in the D1 overview page?

Kavatch•4/25/25, 12:17 PM

Yes correct

DDiego Hey all, just a quick question. I was able to create and populate a FTS5 virtual...

lambrospetrou•4/25/25, 3:30 PM

Every D1 response contains a metameta object that has this information about rows read and written and query duration itself.

lambrospetrou•4/25/25, 3:57 PM

Hey folks, the main fix for the increased internal errorinternal error errors due to D1 read replication is now fully rolled out in production.

Feel free to re-enable replication again, and give us feedback.

Same guideline applies, D1 Read replication is in BETA. Do not enable directly in production without testing in a staging/test environment for a few days to ensure it works fine for your workload.

Read https://developers.cloudflare.com/d1/best-practices/read-replication/ for how to use.

Aajgeiss0702 thank you. Will there be any kind of notification when these fixes are rolled ou...

Milan•4/25/25, 4:03 PM

See Lambros' latest message.

ac•4/25/25, 7:10 PM

How can I copy a D1 database to another D1 database? I've tried using wrangler d1 exportwrangler d1 export to create a .sql file and then wrangler d1 executewrangler d1 execute with that file, but that fails every time due to foreign key constraints.

I know how to do it locally by just running sqlite3 on the sql file and replacing the file, but for remote changes it's confusing why the d1 exports generate invalid SQL for the executeexecute command. Any tips?

ac•4/25/25, 7:12 PM

Before anyone mentions it, yeah the export includes PRAGMA defer_foreign_keys=TRUE;PRAGMA defer_foreign_keys=TRUE; at the top

ac•4/25/25, 7:24 PM

Looks like it works if I change that to PRAGMA foreign_keys = off;PRAGMA foreign_keys = off; and turn off FKs completely for the duration of the file (then on again at the end)

ac•4/25/25, 8:04 PM

Ok, separate question: is there any way to programmatically know what local sqlite files match to which DB bindings for a worker? I have multiple that all persist to the same directory and the sqlite file names are all hashed and don't seem related to the database ID or anything

gruntlord6•4/25/25, 9:21 PM

why not just have a worker batch the contents?

seflanka•4/25/25, 10:27 PM

Is there any ETA on when Cloudflare SQL D1 clusters will be available in Brazil? When can we expect Cloudflare SQL D1 infrastructure to be available in Brazil?

Chaika•4/25/25, 10:48 PM

?crossposting -> workers-and-pages-discussions

SuperHelpflare•4/25/25, 10:48 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Llambrospetrou Hey folks, the main fix for the increased `internal error` errors due to D1 read...

Zack•4/26/25, 12:25 AM

I've just rolled up our production load over the last hour to 100% and so far, zero errors!

Zack•4/26/25, 12:26 AM

Initially had some increased latency but now presumably the database has "warmed up" and we are back to the same latency as on our pre-replica setup!

Zack•4/26/25, 12:27 AM

Going to keep an eye on it, but will probably leave it running for the next 12 hours and check how it looks in the morning from our monitoring

parisholley•4/26/25, 1:53 PM

would enabling read replicas allow us to do an export of the database without blocking requests?

Pparisholley would enabling read replicas allow us to do an export of the database without bl...

lambrospetrou•4/26/25, 2:46 PM

Not yet. We will be working on that soon.

Zack•4/26/25, 7:37 PM

Some more feedback: I reverted back to our original setup after a few hours, though we saw no errors, latencies did increase more than I would have liked. Note that these graph quantiles aren't accurate for their quantile due to the way the metrics are currently aggregated, but the difference between the before and after is accurate and does show a significant shift in our latencies

Zack•4/26/25, 7:37 PM

However we experienced zero errors whatsoever, so that is a significant improvement from before!

Zack•4/26/25, 7:38 PM

I think I asked this question before, but I don't think it was answered. For the replicas, when they startup, do they have to go back to the origin DC for all of the data they need, or just for any recent checkpoints they are missing? I ask this because I think the reason we are getting these latency spikes is for regions where it was currently nighttime and the requests were low

Zack•4/26/25, 7:39 PM

I think for APAC for example, the latency can spike substantially if the replica is currently shutdown as it needs to fetch the data from the source in WEUR

Zack•4/26/25, 7:40 PM

However, if this data was cached near the replica location so it only had to go back "home" for new checkpoints, then I would expect this to not have much impact, especially for our database as it is entirely static, we never insert new rows

ZZack Some more feedback: I reverted back to our original setup after a few hours, tho...

lambrospetrou•4/27/25, 8:02 AM

but the difference between the before and after is accurate and does show a significant shift in our latencies

Where is the difference in this graph? Can you post that instead of the percentiles that you say are wrong? What is the magnitude of the difference you see, 1s, 5s, 500ms?

ZZack I think I asked this question before, but I don't think it was answered. For the...

lambrospetrou•4/27/25, 8:06 AM

The snapshot to boot a replica initially is fetched from the primary's region as far as I remember, but will double check and confirm. Then, once the replica is caught up, updates flow from the primary to all replicas in parallel while they are written to the primary. This should still only add a few hundred ms delay vs if it was in the replica's region, not multiple seconds, unless networking is really bad in those regions (APAC is quite noisy).

Either way, this case is already something we are working on improving/fixing soon, so that you don't need to wait for replicas to boot and they are used only once they are booted. So, these spikes should be eliminated.

Llambrospetrou > but the difference between the before and after is accurate and does show a si...

Zack•4/27/25, 8:15 AM

I can try to get some proper numbers on that, I did break it down by Colo and see some significant spikes. I do have metrics specifically for the D1 call so I can look at pulling those out and getting something useful for you guys probably tomorrow

Zack•4/27/25, 8:18 AM

If it's going back to the primary to pull the entire database I can see that being part of the reason for this. Pulling 2.5GB from WEUR to APAC or OC will surely take more than a few hundred ms. I feel like caching the checkpoints in each region would help significantly there, especially if you immediately boot the database and then sync the changes after. Clients that specify they don't care about any checkpoint could theoretically get the same performance as the primary, if it's cached

Zack•4/27/25, 8:18 AM

I'm sure there's significant complications to doing something like that though that I can't see from the outside

ZZack If it's going back to the primary to pull the entire database I can see that bei...

lambrospetrou•4/27/25, 8:22 AM

Pulling 2.5GB from WEUR to APAC or OC will surely take more than a few hundred ms.

Yeah, but in theory only the TTFB will be the significant difference, again with many caveats here.

Hopefully we will get the non-blocking replica boot soon, so this won't be a problem even without caching the snapshots in all regions.

Zack•4/27/25, 8:23 AM

Yea that would be a significant improvement

Llambrospetrou > Pulling 2.5GB from WEUR to APAC or OC will surely take more than a few hundred...

Zack•4/28/25, 1:47 PM

Hey, so coming back with some more (correct) metrics now.
Firstly, these are significantly improved because we made a change to always query D1, regardless of if the response is cached in the CDN, this seems to keep D1 significantly warmer, and our response times are much better in general than before we added this.
Now, onto the data. As I said before, errors are basically gone now, however we still get latency spikes. I've drilled into this pulling some data split by region, this is specifically metrics around the D1 call from the worker.
As you can see (in this 95th percentile graph), OC is our worst offender here, it occasionally has significant spikes in latency, sometimes going over a minute.
It is currently nighttime there, so we don't have many requests going through for OC, however a minute is still significant.
As we said, the upcoming changes around bypassing the replica when it's booting up would help this for sure, but a minute still seems pretty excessive

Zack•4/28/25, 1:47 PM

Lmk if you want me to pull any other metrics for you

Zack•4/28/25, 1:48 PM

A logarithmic version of the graph so you can see things are generally working normally other from those few OC spikes, one WNAM and one APAC spike

ZZack Hey, so coming back with some more (correct) metrics now. Firstly, these are sig...

lambrospetrou•4/28/25, 2:45 PM

Thanks a lot for the numbers! Yeah, the minute latency is almost always due to very slow network/retries/failures.
We will prioritize the non-blocking replica cold starts which will eliminate all this noise.

Zack•4/28/25, 2:48 PM

Yea makes sense, hopefully you can solve those slow starts too at some point, it feels like caching all that data locally within the DC would help there surely, as you don't have to pull up to 10GB across multiple oceans

I'm seeing lots of worker errors with `Error: D1_ERROR: D1 DB's isolate exceeded its memory limit an

Similar Threads

Similar Threads

Similar Threads