Cloudflare Developers•2y ago

those are problems that we're looking at fixing with dynamic bindings, and change-data-capture

Max (@rozenmd)OP•5/29/24, 8:20 PM

wouldn't make sense to let folks migrate a million dbs in one go imo

MMax (@rozenmd)those are problems that we're looking at fixing with dynamic bindings, and chang...

Hello, I’m Allie!•5/29/24, 8:21 PM

Would the dynamic bindings system support listing all existing DBs? I know that is an issue with DOs where you have to store their IDs somewhere

Hello, I’m Allie!•5/29/24, 8:21 PM

Or have a list of names stored somewhere

Max (@rozenmd)OP•5/29/24, 8:23 PM

still speccing it out - DO not listing instances is also a pain I personally want fixed

Max (@rozenmd)OP•5/29/24, 8:41 PM

we released the .raw().raw() API as an HTTP endpoint: https://developers.cloudflare.com/api/operations/cloudflare-d1-raw-database-query

It fixes the issue where you couldn't JOIN two tables containing columns with the same name

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

MMax (@rozenmd)we released the `.raw()` API as an HTTP endpoint: https://developers.cloudflare....

Isaac McFadyen•5/29/24, 8:43 PM

Ohhhh this is so helpful!

Isaac McFadyen•5/29/24, 8:44 PM

The column names in particular.

Isaac McFadyen•5/29/24, 8:44 PM

Right now I'm doing 2 queries in d1-jdbc-driver (query + get the column schema), this way I can do a query once to get the schema and then match with the cached schema.

Isaac McFadyen•5/29/24, 8:44 PM

I guess I could do that before with the keys... but still handy here.

Kkraustifer https://x.com/tableplus/status/1776611556509253687?s=46&t=KvGal2nXGgMSIuYjfJ6YGQ

Koosh•5/29/24, 10:40 PM

I’ve got table plus, it doesn’t seem to work for one of my tables, program locks up and gives some error that goes off the screen so you need to force close

Avi•5/30/24, 3:35 AM

hey! am i remembering correctly that D1 is internally implemented on top of Durable Objects?

AAvi hey! am i remembering correctly that D1 is internally implemented on top of Dura...

Isaac McFadyen•5/30/24, 3:40 AM

So it used to be, and still is, partially. There were two backends:
Alpha: implemented on Durable Objects using the Durable Objects Storage API. This was quite slow and eventually deprecated and will no longer receive updates.
Beta and Production (what we have now): a whole new SQLite storage engine was implemented on top of Durable Objects, meaning it still uses DOs but the new engine. The SQLite engine is not yet exposed to users but they've said they eventually want to allow DOs to directly access it.

A Twitter thread by the Workers lead engineer on this topic: https://x.com/KentonVarda/status/1659551757796515846

Original message was deleted

Bọc Link VN•5/30/24, 8:22 AM

thank you for sharing

divby0•5/30/24, 10:26 AM

this makes the migrations fail since they are in the wrong order. How does wrangler decide which one to apply first?

divby0•5/30/24, 10:27 AM

like I can change the pattern for naming these, but I thought putting the date in the name was the best way to make it be in the right order automatically

Ddivby0 this makes the migrations fail since they are in the wrong order. How does wrang...

Max (@rozenmd)OP•5/30/24, 11:01 AM

https://github.com/cloudflare/workers-sdk/blob/main/packages%2Fwrangler%2Fsrc%2Fd1%2Fmigrations%2Fapply.tsx#L101

GitHub

workers-sdk/packages/wrangler/src/d1/migrations/apply.tsx at main ·...

Home to Wrangler, the CLI for Cloudflare Workers® - cloudflare/workers-sdk

workers-sdk/packages/wrangler/src/d1/migrations/apply.tsx at main ·...

Max (@rozenmd)OP•5/30/24, 11:01 AM

theres a migration create command that uses the expected naming

MMax (@rozenmd)https://github.com/cloudflare/workers-sdk/blob/main/packages%2Fwrangler%2Fsrc%2F...

divby0•5/30/24, 11:56 AM

not too advanced, but when I read correct, I can simply drop the dashes inbetween the numbers but the "_migration" is okay since it splits there?

Original message was deleted

divby0•5/30/24, 12:00 PM

Got it - okay I have just worked around it temporarily by adding the files into the migrations folder one by one until it's fixed

divby0•5/30/24, 12:00 PM

Thank you all very much!

Original message was deleted

Max (@rozenmd)OP•5/30/24, 12:12 PM

aight i'll cut a PR

Mrinank•5/30/24, 6:06 PM

does D1 binding act like a VPC? Some people suggest putting our databases inside a VPC for security reasons, so that the databases are not accessible through the internet.

Mrinank•5/30/24, 6:11 PM

and can we expose our database_id in git publically?

MMrinank does D1 binding act like a VPC? Some people suggest putting our databases inside...

Hello, I’m Allie!•5/30/24, 6:18 PM

All bindings are only accessible on your account, yes

Hello, I’m Allie!•5/30/24, 6:18 PM

So it is safe to publish to git

Suman!•5/31/24, 7:28 AM

Is anybody aware of this issue? Trying to use d1-dialect causes my tests to fail with-

TypeError: Cannot use require() to import an ES Module.

TypeError: Cannot use require() to import an ES Module.

https://github.com/aidenwallis/kysely-d1/issues/27 . Please help.

GitHub

Using D1Dialect causes worker tests to fail - Cannot use require() ...

Initializing D1Dialect causes my tests to not run as expected. private async initializeDb(db: D1Database) { this.db = new Kysely<Database>({ dialect: new D1Dialect({ database: db }), }); } Ha...

andrew•5/31/24, 7:56 AM

I'm excited about potentially building my next app fully native on CF. My only blocker is d1's storage limits. to work around i'd have to support db/customer. my question is making the migration story and dynamically creating dbs more ergonomic on the roadmap?

Aandrew 👋 I'm excited about potentially building my next app fully native on CF. My onl...

Max (@rozenmd)OP•5/31/24, 8:08 AM

it is indeed on the roadmap - i just (yesterday) drafted the specification for how dynamic bindings and migrations will fit together

Denni•5/31/24, 12:05 PM

Hey all! I'm having some trouble with my D1 database. It looks like it's using a cached result, both using the CLI and in my wrangler and I'm quite unsure why

Denni•5/31/24, 12:05 PM

Denni•5/31/24, 12:05 PM

This is the actual data, as you can see there are total_usages which are much high than what the cli is returning. Am I doing something wrong here? I simply want to get the highest total_usage from the database

Original message was deleted

Denni•5/31/24, 12:06 PM

--remote

--remote

is in there already

Denni•5/31/24, 12:06 PM

My deployed wrangler also returns the same incorrect result

Denni•5/31/24, 12:07 PM

It's also coincidentally the latest result from when I dumped the db yesterday and recreated it (the pack_id column was an INTEGER before, but it's usually above the max_safe_int)

DDenni It's also coincidentally the latest result from when I dumped the db yesterday a...

Max (@rozenmd)OP•5/31/24, 12:40 PM

are they definitely querying the same database id?

DDenni Hey all! I'm having some trouble with my D1 database. It looks like it's using a...

Matt Silverlock•5/31/24, 12:47 PM

wrangler

wrangler

and/or D1 don't cache results (at all). As Max said: can you confirm it's the same database ID here? Your screenshot shows it starting with 50a19e5e50a19e5e - is that the same DB you're looking at in the dash?

MMax (@rozenmd)are they definitely querying the same database id?

Denni•5/31/24, 12:48 PM

They're both using the same wrangler.toml I'd think

DDenni They're both using the same wrangler.toml I'd think

Matt Silverlock•5/31/24, 12:48 PM

Share it with us (redact anything - but IDs are safe to share) to be sure.

Denni•5/31/24, 12:49 PM

[[d1_databases]]
binding = "USAGE_DB"
database_name = "sticker-usage"
database_id = "50a19e5e-a287-41ea-b908-b27ed1cb4787"

[[d1_databases]]
binding = "USAGE_DB"
database_name = "sticker-usage"
database_id = "50a19e5e-a287-41ea-b908-b27ed1cb4787"

Denni•5/31/24, 12:49 PM

Denni•5/31/24, 12:49 PM

Both the same

Denni•5/31/24, 12:49 PM

The wrangler.toml is in the same directory as where I'm running the wrangler commands from

Matt Silverlock•5/31/24, 12:51 PM

There's no caching (concretely, absolutely, in no way is there result caching) , so I'm curious as to why this is showing different data.

Matt Silverlock•5/31/24, 12:51 PM

Are the row counts the same?

Denni•5/31/24, 12:51 PM

Yeah so am I, anything I can try here to make sure the right things are being queried? Some verbose output of the queries

Max (@rozenmd)OP•5/31/24, 12:52 PM

can you try run that query in the console tab?

MMatt Silverlock Are the row counts the same?

Denni•5/31/24, 12:52 PM

They are not

MMax (@rozenmd)can you try run that query in the console tab?

Denni•5/31/24, 12:53 PM

I'm sorry, the console tab? My browser console?

DDenni I'm sorry, the console tab? My browser console?

Matt Silverlock•5/31/24, 12:53 PM

In the D1 dashboard [from your screenshot]

Matt Silverlock•5/31/24, 12:53 PM

those are problems that we're looking at fixing with dynamic bindings, and change-data-capture

Similar Threads

Similar Threads

Similar Threads