Is there any way to make D1 store 8-byte integers? It is currently screwing snowflake IDs. `92233720

Is there any way to make D1 store 8-byte integers? It is currently screwing snowflake IDs.

9223372036854775807

9223372036854775807

9223372036854775807

9223372036854775807 ->

9223372036854776000

9223372036854776000

9223372036854776000

9223372036854776000

BBeyondo Is there any way to make D1 store 8-byte integers? It is currently screwing snow...

lambrospetrou•12/5/24, 9:29 AM

Are you sure it's D1 and not Javascript? JS only has 52-bit precision.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number#number_encoding

Number.parseInt("9223372036854775807")
9223372036854776000

Number.parseInt("9223372036854775807")
9223372036854776000

Store those IDs as strings in D1 SQLite, or as their raw bytes in a BLOB column.

can•12/5/24, 9:47 AM

just a UI thing, when I want to insert some data from the UI, sometimes these buttons blocks the content like this. maybe a smol fix would be awesome

Llambrospetrou Are you sure it's D1 and not Javascript? JS only has 52-bit precision. https://d...

BeyondoOP•12/5/24, 12:23 PM

No, did not use javascript at all. It's a D1 issue. Just execute this query directly for example:

CREATE TABLE test_bigint (value BIGINT);
INSERT INTO test_bigint (value) VALUES (9223372036854775807);

CREATE TABLE test_bigint (value BIGINT);
INSERT INTO test_bigint (value) VALUES (9223372036854775807);

Then run

SELECT * FROM test_bigint;

SELECT * FROM test_bigint;

Output:

BeyondoOP•12/5/24, 12:26 PM

But yeah, you are right. I'd probably just go for strings and a whole other algorithm than Snowflake anyways. I simply thought D1 supported big integers "internally" only not in the JS API (or so it said in the docs)—I could've misinterpreted that.

BBeyondo No, did not use javascript at all. It's a D1 issue. Just execute this query dire...

lambrospetrou•12/5/24, 12:26 PM

The output you see is in the UI right? In the website, so it's parsing the number in Javascript. The D1 responses are also parsed in JS so this applies there too.

Llambrospetrou The output you see is in the UI right? In the website, so it's parsing the numbe...

BeyondoOP•12/5/24, 12:27 PM

Ohh that makes sense! I can't just put a number like that directly in the query then.. thank you for correcting me

justaname•12/6/24, 3:50 AM

does cloudflare themselves use d1 database?

Jjustaname does cloudflare themselves use d1 database?

Max (@rozenmd)•12/6/24, 7:28 AM

yes

nanfeng•12/6/24, 11:38 AM

Getting this error on simple query: Error: D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

nanfeng•12/6/24, 11:39 AM

Simple sql query returning 1 row from 10 rows is taking 748.00 ms, the worker's placement is changed to smart and spamming get request every 5s for more than 24 hours. the latency is still crazy high

Nnanfeng Getting this error on simple query: Error: D1_ERROR: D1 DB storage operation exc...

Isaac McFadyen•12/6/24, 1:32 PM

Mind sending the ID of your DB (you can put it in here, it's not a secret because someone would need your account login to use it)?

nanfeng•12/6/24, 1:33 PM

e7459e84-1172-4d65-9f2a-3f52cae573a2

Isaac McFadyen•12/6/24, 1:33 PM

Thanks

Isaac McFadyen•12/6/24, 1:42 PM

I've raised this internally. No timeline on a fix but assuming it's the issue I'm thinking of, it's often a simple fix.

Kiki Y•12/6/24, 4:27 PM

Hi team, this is Kiki from Cloudflare, let me PM the impacted user to get a ticket for further investigation.

Kiki Y•12/6/24, 5:12 PM

update, already created the ticket, will have further follow up through ticket then.

MMax (@rozenmd)yes

justaname•12/7/24, 4:18 AM

what is it used in at cloudflare or is it most things or also do they use the multi tenant way or?

Jadugar_Jaggu•12/7/24, 6:28 AM

Hey, why is my D1 database so slow? It's literally taking 20–30 seconds to execute an UPDATE query! sometime even minutes

Jjustaname what is it used in at cloudflare or is it most things or also do they use the mu...

James•12/7/24, 7:47 AM

Radar uses it for ratings: https://blog.cloudflare.com/radar-data-explorer-ai-assistant/
The cf docs just launched something similar too and I’d guess that’s stored in D1 as well

The Cloudflare Blog

Network trends and natural language: Cloudflare Radar’s new Data Ex...

The Cloudflare Radar Data Explorer provides a simple Web-based interface to build more complex API queries, including comparisons and filters, and visualize the results. The accompanying AI Assistant translates a user’s natural language statements or questions into the appropriate Radar API calls.

justaname•12/7/24, 8:10 AM

oh cool to see thank u

limzykenneth•12/7/24, 5:37 PM

Is something going on for D1 at the moment as I'm seeing gateway timeout when running queries on my D1 database at the moment, or is it just me?

kamama•12/8/24, 2:31 PM

Is D1 support SOUNDEX?

majestic•12/9/24, 2:24 AM

Hello, can i select d1 on specific region ?

sdayman•12/9/24, 2:28 AM

You can provide a Location Hint when you create the database:
https://developers.cloudflare.com/d1/configuration/data-location/#provide-a-location-hint

kamama•12/9/24, 2:39 AM

https://github.com/cloudflare/workers-sdk/issues/5664

May I know if cloudflare have any timeline for support this ?

GitHub

🚀 Feature Request: Support load_extensions (for spellfix1) · Issue ...

Describe the solution Currently, we cannot use D1 since we need the usage of editdist3 (levenshtein distance) which is included in the spellfix1 extension. However, when trying to run SELECT load_e...

Kkamama https://github.com/cloudflare/workers-sdk/issues/5664 May I know if cloudflare ...

Isaac McFadyen•12/9/24, 3:13 AM

This is because of security concerns and so I'd tentatively say it's not going to be supported at all.

Isaac McFadyen•12/9/24, 3:13 AM

By allowing users to load their own extensions it opens D1 up to a whole host of security attacks.

kamama•12/9/24, 3:16 AM

Spellfix1 is a loadable extension, and it’s already pre compiled?

kamama•12/9/24, 3:16 AM

It’s not enable by default only, how come it brings security concern

Isaac McFadyen•12/9/24, 3:19 AM

Not sure but the SQLite wiki specifically says that extension loading is disabled by default for security reasons:
https://www.sqlite.org/loadext.html

For security reasons, extension loading is turned off by default.

SakuraFC•12/9/24, 4:04 AM

I am not sure if this has been answered before. Given the per-DB size limit and recommendation to shard your data into separate databases, how would one go about it in practice. Let’s say I wanted to have a separate DB per customer, how would I create one on the fly and bind it to a worker? Or am I misreading something?

kamama•12/9/24, 4:35 AM

D1 API to create DB
PUT project config to update bindings

kamama•12/9/24, 4:35 AM

But not sure how to make it work for existing alive worker

Kkamama But not sure how to make it work for existing alive worker

Isaac McFadyen•12/9/24, 4:45 AM

Currently you can't. Dynamic bindings are on the roadmap.

SSakuraFC I am not sure if this has been answered before. Given the per-DB size limit and ...

Larry•12/9/24, 12:28 PM

Have you explored using Cloudflare Durable Objects (DOs) with the SQLite API? They provide a workaround for the current lack of dynamic binding for D1. One DO per customer is a common pattern.

LLarry Have you explored using Cloudflare Durable Objects (DOs) with the SQLite API? Th...

SakuraFC•12/9/24, 12:42 PM

Thanks, I was looking into that, but so far the limit is 1 GB and not sure how to ensure data backup yet

IIsaac McFadyen Currently you can't. Dynamic bindings are on the roadmap.

o7•12/9/24, 4:58 PM

Do we have a very rough eta for when this might arrive?

SSakuraFC Thanks, I was looking into that, but so far the limit is 1 GB and not sure how t...

Larry•12/9/24, 7:27 PM

The 1GB limit is while in Beta. It will match D1's 10GB once it goes GA.

SSakuraFC Thanks, I was looking into that, but so far the limit is 1 GB and not sure how t...

Larry•12/9/24, 7:28 PM

Offline backup is a lot harder. You will just have to trust Cloudflares time travel capability, which is not super reassuring, or roll your own.

Ian•12/10/24, 1:29 AM

I also want to know ETA for D1 dynamic binding, we've been asking for more than a year now and we get the same answer every time (soon

without any timeframe approximation).

IIan I also want to know ETA for D1 dynamic binding, we've been asking for more than ...

Hello, I’m Allie!•12/10/24, 1:58 AM

Just wondering, what would the D1 Binding give you that DO SQLite doesn't give you?

HHello, I’m Allie!Just wondering, what would the D1 Binding give you that DO SQLite doesn't give y...

Ian•12/10/24, 2:04 AM

pricing is drastically different on a read-heavy workflow (specially indexed), also includes a much higher monthly allowance on usage for workers paid

Burrito•12/10/24, 2:05 AM

I'm under the impression that DO SQLite has the same pricing as D1? Unless you are talking about the extra cost of DO itself.

BBurrito I'm under the impression that DO SQLite has the same pricing as D1? Unless you a...

Ian•12/10/24, 2:13 AM

DO SQLite AFAIK uses DO pricing, each read and write

Ian•12/10/24, 2:14 AM

Do let me know if this is not true

Burrito•12/10/24, 2:14 AM

There are two pricing models for DO, there's the KV storage backend and the SQLite storage backend. For the SQLite storage backend, pricing model is the same as D1.

Ian•12/10/24, 2:16 AM

Thanks, I had to do a full search of the docs to find it
https://developers.cloudflare.com/durable-objects/best-practices/access-durable-objects-storage/#sqlite-storage-backend

Ian•12/10/24, 2:17 AM

If this is the case, then I have nothing more than to wait for 10GB storage limit per DB

sor4chi/monica•12/10/24, 3:52 AM

Hello. I have a question about D1.

Even for a very simple query like “SELECT 1”, the database latency takes around 500ms. This number is confirmed using serverTiming, so I’m certain it’s not caused by other factors outside D1.

Additionally, I have confirmed that several people in the X community are experiencing the same issue. I tested this by sending requests to Workers with D1 binding from the Tokyo area.

If anyone else can reproduce this, or if you know the cause or any solutions, I would greatly appreciate your input.

Ssor4chi/monica Hello. I have a question about D1. Even for a very simple query like “SELECT 1”...

lambrospetrou•12/10/24, 9:18 AM

Where is your database located? Usually network is what takes up the whole time when querying D1.
The meta.durationmeta.duration field should tell you how much the actual SQL query takes when running inside your database.
Everything else is networking to get from your Worker location to your D1 database location.

This is not to say there aren't things to improve internally (there are!!), but if worker and database are far away, then hundreds of ms latency is to be expected.

Is there any way to make D1 store 8-byte integers? It is currently screwing snowflake IDs. `92233720

Similar Threads

Similar Threads

Similar Threads