Hello everyone, Quick question. Is D1 a suitable backend database for a financial wallet web appli

Hello everyone,

Quick question. Is D1 a suitable backend database for a financial wallet web application? i.e. ensuring consistency across users funds balances / withdrawing funds and depositing funds reflect consistently across all users of the wallet app. Is D1 sufficient for this use case? or will i need a Postgres db for example?

Thanks,

JJames For what specifically? You'll find some of D1 source in the `workerd` repo like ...

Kotkoroid•11/23/24, 7:40 PM

Specifically for the D1 engine. Afaik it's an extension of SQLite engine.

TTOPGUN Hello everyone, Quick question. Is D1 a suitable backend database for a financ...

Larry•11/23/24, 8:53 PM

From a consistency perspective, D1 is on par with Postgres. Reliability and recoverability are more a function of how each is hosted and Cloudflare is well above average on this front. The biggest difference is scaling both in size and throughput. D1 maxes out at 10GB. Postgres DBs can be TBs. Similarly, D1 also has significant throughput scaling limitations compared to Postgres and I suspect that will continue even after D1 is upgraded to support read replicas.

LLarry From a consistency perspective, D1 is on par with Postgres. Reliability and reco...

João Castro•11/23/24, 9:30 PM

I remember reading a message about "unlimited read throughput" once read replicas arrive. But writes would still have the same throughput limitations as it has today.

JJoão Castro I remember reading a message about "unlimited read throughput" once read replica...

Larry•11/23/24, 11:01 PM

That's good news. I hadn't read that.

ella•11/24/24, 2:30 PM

is d1 down? i keep getting auth errors

ella•11/24/24, 2:32 PM

@Walshy ?

Flare•11/24/24, 2:35 PM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

Vlady•11/24/24, 3:07 PM

Hi y'all, what's the latest news on transaction support?
I'm using the Drizzle ORM and transactions don't work there: https://github.com/drizzle-team/drizzle-orm/issues/2463

GitHub

[BUG]: Cloudflare D1 transaction not supported · Issue #2463 · driz...

What version of drizzle-orm are you using? 0.31.1 What version of drizzle-kit are you using? 0.22.2 Describe the Bug When trying to execute a transacction with cloudflare d1 adapter it throws the f...

elpupper•11/24/24, 3:36 PM

how do i force a schema change? I added autoincrement like a noob

maco6517•11/24/24, 6:12 PM

helpme

Mmaco6517 helpme

Isaac McFadyen•11/24/24, 6:47 PM

This is the Cloudflare D1 channel, unrelated to MySQL. I'd recommend positing in #off-topic, but at the same time, there's not much we can do to help with a single screenshot of a MySQL process using a lot of resources.

rumata•11/25/24, 12:55 AM

hi. i've been trying to make d1 work in rust with bigint (i64) data types, and i keep getting this error: Error: Error: D1_TYPE_ERROR: Type 'bigint' not supported for value '300' - Cause: Error: D1_TYPE_ERROR: Type 'bigint' not supported for value '300'. what am i missing?

Rrumata hi. i've been trying to make d1 work in rust with bigint (i64) data types, and i...

lambrospetrou•11/25/24, 9:51 AM

D1 is based on SQLite, so these are the types you can use: https://www.sqlite.org/datatype3.html
Not sure how Rust is involved in your code, but

bigint

bigint

is not a valid SQLite type and gets converted to one of the native ones, in this case

INTEGER

INTEGER

.

How are you using Rust, are you using the D1 Worker binding with Rust WASM, or something else?
That error is coming from the D1 Worker binding, so it expects JS types.

rumata•11/25/24, 12:30 PM

yes, i am using d1 worker with rust and i have a i64 variable, i am trying to bind to an Integer column when i get this error. Here is the code and schema:
Drop Table Test;
CREATE TABLE IF NOT EXISTS Test (
Id TEXT PRIMARY KEY NOT NULL,
Value INTEGER NOT NULL
);

rumata•11/25/24, 12:30 PM

let d1 = ctx.env.d1("DB")?;
let statement = d1.prepare("INSERT INTO Test (Id, Value) VALUES (?1, ?2)");
let v: i64 = 300;
let query = statement.bind(&["abc".into(),
v.into()])?;
let result = query.run().await?;
console_log!("result: {:?}", result.success());

rumata•11/25/24, 12:51 PM

i think i figured this out - i have to convert everything to string on rust side - and then despite things being strings, d1 interprets them correctly as integer of varying size.

Rrumata i think i figured this out - i have to convert everything to string on rust side...

lambrospetrou•11/25/24, 1:13 PM

Are you sure they are written as integers though? SQLite non-strict tables accept anything you give them regardless of the column type. I would suggest converting the i64 into a normal integer before passed to the statement bind call. Maybe this works, and you will keep them as integers.

Christian•11/26/24, 8:58 PM

Super weird issue but could it be that the Cloudflare D1 export via

npx wrangler d1 export dharmaradio --remote --output=dump.sql

npx wrangler d1 export dharmaradio --remote --output=dump.sql

creates a dump which can't be imported again via

npx wrangler d1 execute dharmaradio --remote --file=dump.sql

npx wrangler d1 execute dharmaradio --remote --file=dump.sql

.

The first issue is that datetime fields aren't put under quotes. After fixing that via multi-cursor editing I'm getting the following error

{"D1_RESET_DO":true}

{"D1_RESET_DO":true}

.

I know D1 is still new but I would expect that I could import the dump I exported via it.

Original message was deleted

Christian•11/26/24, 9:12 PM

40mb but that doesn't explain why the export can't be imported

Original message was deleted

Christian•11/26/24, 9:23 PM

No, it first fails on parsing the sql because of the missing quotes around the dates. Once i fixed that it succeeds parsing the sql and then breaks with the d1_reset_do error.

I wrote a script to chunk the inserts in the dump so hopefully that works out but the export 100% has some issue

GooningChatter•11/26/24, 10:27 PM

(In this screenshot

ENAM_OBJECTS

ENAM_OBJECTS

is an empty database in a closer pop than

ENAM_USERS

ENAM_USERS

. Same with the

WNAM

WNAM

databases)

GooningChatter•11/26/24, 10:44 PM

It's especially apparent when on average it's this drastic

dev•11/27/24, 12:26 AM

I need to specify where my D1 database will be located. As in, it needs to be in Canada. Is it possible? I can only see location hints but I don't think that's enough due to legal reasons

Ddev I need to specify where my D1 database will be located. As in, it needs to be in...

Isaac McFadyen•11/27/24, 12:42 AM

Correct, location hints won't be enough if it's for compliance. And I don't believe D1 supports Jurisdiction Hints like R2 (and even if it did, there's no Canada jurisdiction at the moment).

IIsaac McFadyen Correct, location hints won't be enough if it's for compliance. And I don't beli...

dev•11/27/24, 2:24 AM

That’s very unfortunate. I love the d1 experience, but yeah… compliance is compliance

DDeleted User How do I make sure 2 databases in the same region spawn in the same colo? For ex...

lambrospetrou•11/27/24, 11:36 AM

No, you cannot control more fine-grained locations than regional location hints.

juanferreras•11/27/24, 7:24 PM

I was using nanoid on a small pet project and wanted to change the id lengths to start pretty small (probably 4 characters), and start increasing as we get more rows populated (to avoid collissions). Mainly to make some of the shareable URLs nicer for first users too

generally speaking, I wouldn't be concerned on doing a

COUNT(id)

COUNT(id)

every X minutes and storing it somewhere like KV (from a database workload nor a performance standpoint). But I've just realized that'd be an EXTREMELY bad idea with SQLLite and D1 row-based pricing right?

juanferreras•11/27/24, 7:54 PM

aah that's a great solution! I think I can just setup triggers to keep those +1 automatic

juanferreras•11/27/24, 7:54 PM

thanks, Leo!

sz19•11/28/24, 7:16 AM

Is there anyway to find out which table is being accessed the most?

Ssz19 Is there anyway to find out which table is being accessed the most?

lambrospetrou•11/28/24, 11:43 AM

You can find some query metrics for your usage as described in https://developers.cloudflare.com/d1/observability/metrics-analytics/
I guess the most relevant to what you asked is seeing the top queries and checking the tables accessed.

lucasp•11/28/24, 1:53 PM

Hello, I'm using Prisma with Hono and Workers. I've noticed that when I use Prisma, it takes more time to execute queries compared to using db.prepare. Is that normal?

Dean•11/28/24, 9:42 PM

is cf aware of random issues still with D1? Getting internal error's every 5 - 6 hours. Error processing database Error: Failed to execute query: 400 Bad Request - {"errors":[{"code":7500,"message":"Internal error while starting up D1 DB storage caused object to be reset."}],"success":false,"messages":[],"result":[]}

sz19•11/29/24, 8:44 AM

PRO TIP: Index your tables

DDean is cf aware of random issues still with D1? Getting internal error's every 5 - 6...

sz19•11/29/24, 8:49 AM

I also still get D1_ERROR: Network connection lost. from time to time

DDean is cf aware of random issues still with D1? Getting internal error's every 5 - 6...

DaniFoldi•11/29/24, 2:38 PM

@Dean @sz19 @Shannor can you share database IDs of the dbs where you're experiencing the issues?

DDaniFoldi @Dean @sz19 @Shannor can you share database IDs of the dbs where you're experien...

Dean•11/29/24, 2:47 PM

2206f7ed-1476-40a0-9758-f1bc3d47e5ad for the last 12 hours it’s been okay, but it comes and goes. I have a cf worker that connects to it every hour to do auditing and it fails

D Trombett•11/29/24, 3:10 PM

Hi, sometimes I'm getting a

D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.

. This happens rarely, but has been happening for weeks and when it happens no request is able to query the database. It seems like it randomly freezes for 30s and then returns that error
Database ID:

8a298455-ac11-4e0c-b160-4db807cf3572

8a298455-ac11-4e0c-b160-4db807cf3572

DD Trombett Hi, sometimes I'm getting a `D1_ERROR: D1 DB storage operation exceeded timeout ...

Isaac McFadyen•11/29/24, 3:11 PM

Can you give the database ID please (it's not a secret/can't be used except for on your own account so you can share it here)?

D Trombett•11/29/24, 3:12 PM

sure

Isaac McFadyen•11/29/24, 3:12 PM

Thanks.

Original message was deleted

Isaac McFadyen•11/29/24, 3:13 PM

Not sure there's an immediate fix but we're surfacing these errors to the team so hopefully there's some communication at some point.

D Trombett•11/29/24, 3:14 PM

Don't worry, I'm using D1 just for a small hobby project so it's nothing critical

Shannor•11/29/24, 3:38 PM

Hey @DaniFoldi and @Isaac McFadyen the Database ID is 7356accc-d066-45c4-a577-9b9009c822eb. We use the DB pretty often at my company so it's definitely a weird error though sporadic. We're not able to reproduce it on purpse and it also seems like we get the error but the DB action still happens.

James•11/29/24, 3:47 PM

These kind of sporadic errors are unfortunately par for the course with D1. Here's some stats from an active DB for example: https://canary.discord.com/channels/595317990191398933/992060581832032316/1300948096964100097
The best you can do is retry queries until they work, sadly, but I wish I had better news for you.

SShannor Hey @DaniFoldi and @Isaac McFadyen the Database ID is 7356accc-d066-45c4-a577-...

Isaac McFadyen•11/29/24, 3:49 PM

it also seems like we get the error but the DB action still happens

This part seems incredibly concerning...?

Isaac McFadyen•11/29/24, 3:50 PM

So to confirm, you're making a query and it's succeeding in doing whatever the query is supposed to, but you're still getting an error?

JJames These kind of sporadic errors are unfortunately par for the course with D1. Here...

James•11/29/24, 4:29 PM

From the D1 team: "[the user should] retry their query and unless there is an underlying issue with the database, it should eventually work". That's the best we've got for now, sorry folks.

Kotkoroid•11/29/24, 6:41 PM

Hey, does D1 engine has public sources?

Hello everyone, Quick question. Is D1 a suitable backend database for a financial wallet web appli

Similar Threads

Similar Threads

Similar Threads