Getting this error on simple query: Error: D1_ERROR: D1 DB storage operation exceeded timeout which

Getting this error on simple query: Error: D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

nanfengOP•12/6/24, 11:39 AM

Simple sql query returning 1 row from 10 rows is taking 748.00 ms, the worker's placement is changed to smart and spamming get request every 5s for more than 24 hours. the latency is still crazy high

Nnanfeng Getting this error on simple query: Error: D1_ERROR: D1 DB storage operation exc...

Isaac McFadyen•12/6/24, 1:32 PM

Mind sending the ID of your DB (you can put it in here, it's not a secret because someone would need your account login to use it)?

nanfengOP•12/6/24, 1:33 PM

e7459e84-1172-4d65-9f2a-3f52cae573a2

Isaac McFadyen•12/6/24, 1:33 PM

Thanks

Isaac McFadyen•12/6/24, 1:42 PM

I've raised this internally. No timeline on a fix but assuming it's the issue I'm thinking of, it's often a simple fix.

Kiki Y•12/6/24, 4:27 PM

Hi team, this is Kiki from Cloudflare, let me PM the impacted user to get a ticket for further investigation.

Kiki Y•12/6/24, 5:12 PM

update, already created the ticket, will have further follow up through ticket then.

MMax (@rozenmd)yes

justaname•12/7/24, 4:18 AM

what is it used in at cloudflare or is it most things or also do they use the multi tenant way or?

Jadugar_Jaggu•12/7/24, 6:28 AM

Hey, why is my D1 database so slow? It's literally taking 20–30 seconds to execute an UPDATE query! sometime even minutes

Jjustaname what is it used in at cloudflare or is it most things or also do they use the mu...

James•12/7/24, 7:47 AM

Radar uses it for ratings: https://blog.cloudflare.com/radar-data-explorer-ai-assistant/
The cf docs just launched something similar too and I’d guess that’s stored in D1 as well

The Cloudflare Blog

Network trends and natural language: Cloudflare Radar’s new Data Ex...

The Cloudflare Radar Data Explorer provides a simple Web-based interface to build more complex API queries, including comparisons and filters, and visualize the results. The accompanying AI Assistant translates a user’s natural language statements or questions into the appropriate Radar API calls.

justaname•12/7/24, 8:10 AM

oh cool to see thank u

limzykenneth•12/7/24, 5:37 PM

Is something going on for D1 at the moment as I'm seeing gateway timeout when running queries on my D1 database at the moment, or is it just me?

kamama•12/8/24, 2:31 PM

Is D1 support SOUNDEX?

majestic•12/9/24, 2:24 AM

Hello, can i select d1 on specific region ?

sdayman•12/9/24, 2:28 AM

You can provide a Location Hint when you create the database:
https://developers.cloudflare.com/d1/configuration/data-location/#provide-a-location-hint

kamama•12/9/24, 2:39 AM

https://github.com/cloudflare/workers-sdk/issues/5664

May I know if cloudflare have any timeline for support this ?

GitHub

🚀 Feature Request: Support load_extensions (for spellfix1) · Issue ...

Describe the solution Currently, we cannot use D1 since we need the usage of editdist3 (levenshtein distance) which is included in the spellfix1 extension. However, when trying to run SELECT load_e...

Kkamama https://github.com/cloudflare/workers-sdk/issues/5664 May I know if cloudflare ...

Isaac McFadyen•12/9/24, 3:13 AM

This is because of security concerns and so I'd tentatively say it's not going to be supported at all.

Isaac McFadyen•12/9/24, 3:13 AM

By allowing users to load their own extensions it opens D1 up to a whole host of security attacks.

kamama•12/9/24, 3:16 AM

Spellfix1 is a loadable extension, and it’s already pre compiled?

kamama•12/9/24, 3:16 AM

It’s not enable by default only, how come it brings security concern

Isaac McFadyen•12/9/24, 3:19 AM

Not sure but the SQLite wiki specifically says that extension loading is disabled by default for security reasons:
https://www.sqlite.org/loadext.html

For security reasons, extension loading is turned off by default.

SakuraFC•12/9/24, 4:04 AM

I am not sure if this has been answered before. Given the per-DB size limit and recommendation to shard your data into separate databases, how would one go about it in practice. Let’s say I wanted to have a separate DB per customer, how would I create one on the fly and bind it to a worker? Or am I misreading something?

kamama•12/9/24, 4:35 AM

D1 API to create DB
PUT project config to update bindings

kamama•12/9/24, 4:35 AM

But not sure how to make it work for existing alive worker

Kkamama But not sure how to make it work for existing alive worker

Isaac McFadyen•12/9/24, 4:45 AM

Currently you can't. Dynamic bindings are on the roadmap.

SSakuraFC I am not sure if this has been answered before. Given the per-DB size limit and ...

Larry•12/9/24, 12:28 PM

Have you explored using Cloudflare Durable Objects (DOs) with the SQLite API? They provide a workaround for the current lack of dynamic binding for D1. One DO per customer is a common pattern.

LLarry Have you explored using Cloudflare Durable Objects (DOs) with the SQLite API? Th...

SakuraFC•12/9/24, 12:42 PM

Thanks, I was looking into that, but so far the limit is 1 GB and not sure how to ensure data backup yet

IIsaac McFadyen Currently you can't. Dynamic bindings are on the roadmap.

o7•12/9/24, 4:58 PM

Do we have a very rough eta for when this might arrive?

SSakuraFC Thanks, I was looking into that, but so far the limit is 1 GB and not sure how t...

Larry•12/9/24, 7:27 PM

The 1GB limit is while in Beta. It will match D1's 10GB once it goes GA.

SSakuraFC Thanks, I was looking into that, but so far the limit is 1 GB and not sure how t...

Larry•12/9/24, 7:28 PM

Offline backup is a lot harder. You will just have to trust Cloudflares time travel capability, which is not super reassuring, or roll your own.

Ian•12/10/24, 1:29 AM

I also want to know ETA for D1 dynamic binding, we've been asking for more than a year now and we get the same answer every time (soon

without any timeframe approximation).

IIan I also want to know ETA for D1 dynamic binding, we've been asking for more than ...

Hello, I’m Allie!•12/10/24, 1:58 AM

Just wondering, what would the D1 Binding give you that DO SQLite doesn't give you?

HHello, I’m Allie!Just wondering, what would the D1 Binding give you that DO SQLite doesn't give y...

Ian•12/10/24, 2:04 AM

pricing is drastically different on a read-heavy workflow (specially indexed), also includes a much higher monthly allowance on usage for workers paid

Burrito•12/10/24, 2:05 AM

I'm under the impression that DO SQLite has the same pricing as D1? Unless you are talking about the extra cost of DO itself.

BBurrito I'm under the impression that DO SQLite has the same pricing as D1? Unless you a...

Ian•12/10/24, 2:13 AM

DO SQLite AFAIK uses DO pricing, each read and write

Ian•12/10/24, 2:14 AM

Do let me know if this is not true

Burrito•12/10/24, 2:14 AM

There are two pricing models for DO, there's the KV storage backend and the SQLite storage backend. For the SQLite storage backend, pricing model is the same as D1.

Ian•12/10/24, 2:16 AM

Thanks, I had to do a full search of the docs to find it
https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-storage/#sqlite-storage-backend

Ian•12/10/24, 2:17 AM

If this is the case, then I have nothing more than to wait for 10GB storage limit per DB

sor4chi/monica•12/10/24, 3:52 AM

Hello. I have a question about D1.

Even for a very simple query like “SELECT 1”, the database latency takes around 500ms. This number is confirmed using serverTiming, so I’m certain it’s not caused by other factors outside D1.

Additionally, I have confirmed that several people in the X community are experiencing the same issue. I tested this by sending requests to Workers with D1 binding from the Tokyo area.

If anyone else can reproduce this, or if you know the cause or any solutions, I would greatly appreciate your input.

Ssor4chi/monica Hello. I have a question about D1. Even for a very simple query like “SELECT 1”...

lambrospetrou•12/10/24, 9:18 AM

Where is your database located? Usually network is what takes up the whole time when querying D1.
The

meta.duration

meta.duration

field should tell you how much the actual SQL query takes when running inside your database.
Everything else is networking to get from your Worker location to your D1 database location.

This is not to say there aren't things to improve internally (there are!!), but if worker and database are far away, then hundreds of ms latency is to be expected.

sor4chi/monica•12/10/24, 10:18 AM

Hi, @lambrospetrou
Indeed, when fetching the meta, it appears to be the case. This suggests that D1 itself is not hopelessly slow.
Does this mean that D1’s location selection strategy is the issue?
As a user, is there any way to intentionally select the location other?
Smart placement didn't work.

Original message was deleted

sor4chi/monica•12/10/24, 10:28 AM

Thank you for the hint.
I recreated the database with

--location=apac

--location=apac

, but the issue persists.

Ssor4chi/monica Thank you for the hint. I recreated the database with `--location=apac`, but the...

lambrospetrou•12/10/24, 10:34 AM

Unfortunately, region-wise location hints are the only way to influence its location at the moment. In certain regions this is still a very wide area though, indeed. For example, APAC can be anywhere between Singapore and Osaka (https://where.durableobjects.live/region/apac).
I can't promise anything on this yet, but it's something we are thinking on how to improve.
Read replication will definitely help with reads on this depending on where your queries originate, but writes are going to still be impacted.

For now, if your servers/workers are in a single location (e.g. Tokyo), try creating one a few times and see if you get one close to you.

Original message was deleted

sor4chi/monica•12/10/24, 10:40 AM

here

Llambrospetrou Unfortunately, region-wise location hints are the only way to influence its loca...

sor4chi/monica•12/10/24, 10:41 AM

After hitting the API about 10 times, it suddenly became faster.
Does this mean that the location of the Durable Object hosting D1 has moved?

Ssor4chi/monica After hitting the API about 10 times, it suddenly became faster. Does this mean ...

lambrospetrou•12/10/24, 10:42 AM

After creation the database doesn't move no. The first query is always going to be slow since you are cold starting your database, its location being cached in Cloudflare's CDN etc.

Llambrospetrou After creation the database doesn't move no. The first query is always going to ...

sor4chi/monica•12/10/24, 10:44 AM

Thank you, I understand now.
I apologize if my understanding is incorrect, but I had heard that D1 delivers “read replicas to the edge,” so I assumed that READ operations would be reasonably fast regardless of the location.

Ssor4chi/monica Thank you, I understand now. I apologize if my understanding is incorrect, but I...

lambrospetrou•12/10/24, 10:54 AM

Read replication is coming soon yes, but it's not rolled out yet.
Even with read replication though, you are still going to go over the network on a different machine, or location even, so it's not going to be a single-digit millisecond solution either (it cannot be due to physics!).

o7•12/10/24, 11:10 AM

if i run a

count + 1

count + 1

update query in a batch statement 1000 times, will it end up being 1000 at the end? ie batches are transactional?

Getting this error on simple query: Error: D1_ERROR: D1 DB storage operation exceeded timeout which

Similar Threads

Similar Threads

Similar Threads