also https://github.com/nora-soderlund/cloudflare-dynamic-bindings might be of interest

MMax (@rozenmd)also https://github.com/nora-soderlund/cloudflare-dynamic-bindings might be of i...

marko.333•7/20/24, 11:47 AM

i'm not sure i would call that github project "dynamic" (maybe "cloudflare-gen-bindings")? Regardless - cool hack.
I wish there was an open discussions page to discuss requests and future direction?
Besides being able to dynamically create resources, it would also be nice to build some higher-level abstractions (at least for common usecases), to help simplify the management.
For example, it would be great if there was some form of higher-level "sharding" support that only required a resource to be configured with a "sharding key" (Example: A D1 "foobar" db shared by "customer_id"). That resource (D1, or Queue, or ...) would then require the shard-key to operate on it.
Under the covers Cloudflare would then manage "foobar" by appropriately creating a D1 instance when a new key is referenced, or finding/binding the instance when it does exist, and Migrations could also be done across the entire "foobar" cluster?

Mmarko.333 i'm not sure i would call that github project "dynamic" (maybe "cloudflare-gen-b...

Max (@rozenmd)OP•7/20/24, 11:53 AM

that's roughly how we're designing d1 dynamic bindings

Max (@rozenmd)OP•7/20/24, 11:58 AM

the migrations story becomes "fun" with that though, since you can now make potentially millions of DBs, we don't want migrations to simultaneously wake up millions of DBs

MMax (@rozenmd)that's roughly how we're designing d1 dynamic bindings

marko.333•7/20/24, 1:19 PM

OK, great to know that more coolness is on the way.
To add another dimension of complexity - hopefully there is also some form of integration into "Gradual Deployments"?
https://developers.cloudflare.com/workers/configuration/versions-and-deployments/gradual-deployments/
Maybe that could be part of the story to ease the cost of migrations?
Btw, is there similar work in-progress to support dynamic binding (and sharding) for Queues?
Thx Max.

Cory•7/20/24, 8:51 PM

ARe there any plans to be able to subscribe to updates?

JoeDoe•7/21/24, 6:30 AM

Is there some client library or something that I can use to access D1 database remotely, but still use the local one for testing without having to write double queries for the remote and local one?

mohitgangrade•7/21/24, 7:09 AM

[ERROR] not authorized: SQLITE_AUTH

I've even tried creating a custom token and uploading the SQL file using the REST API. I keep getting this error after the file gets done uploading.

mohitgangrade•7/21/24, 7:09 AM

mohitgangrade•7/21/24, 7:09 AM

I get this error even in the Cloudflare Dashboard

kaushal•7/21/24, 12:24 PM

hello

kaushal•7/21/24, 12:24 PM

anybody here ?

Flare•7/21/24, 12:32 PM

https://dontasktoask.com/

Don't ask to ask, just ask

Gerbuuun•7/21/24, 8:50 PM

You could query the

sqlite_sequence

sqlite_sequence

table but like Leo said you can get race conditions

Zeeshan•7/22/24, 5:09 AM

Hi folks. i have a bunch of data in a csv file which i would like to dump into my table i d1 db. Which is the best way to do this? Seems like d1 doesn't support csv yet (cmiiw)

Original message was deleted

JoeDoe•7/22/24, 7:40 AM

what dat

Original message was deleted

JoeDoe•7/22/24, 7:42 AM

this can connect to D1 database remotely?

Phaze•7/23/24, 1:45 PM

Heyo, I'm trying to select a map by an id but some of the map ids are all numbers, and something keeps trying to format the id, that is a string, into a float. I can't work out if this is a D1 issue or an issue with workers-rs

https://cdn.phaz.uk/i/Code_5UINmeAWon.png https://cdn.phaz.uk/i/Code_wAffgOTnIe.png

Help would be greatly appreciated :)

PPhaze Heyo, I'm trying to select a map by an id but some of the map ids are all number...

Phaze•7/23/24, 5:22 PM

I'm still getting this issue, even with other parts of my code, it's set to string everywhere yet it's still trying to convert it to a float

niconiahi•7/23/24, 5:43 PM

are there any plans on exposing a connection string to control the product d1 database? I use DBeaver and far far more superior than the db visualizer that you guys have

has there been any conversations around this? it's kind of an obvious move IMO as there is so much tooling already to be used that is better to diagnose a db

it takes 5 more times to do whatever i have to do on the d1 db viewer than DBeaver

Nniconiahi are there any plans on exposing a connection string to control the product d1 da...

Chaika•7/23/24, 5:51 PM

what would that connection string be of? D1 is SQlite, there is no remote sqlite protocol

Chaika•7/23/24, 5:52 PM

You can use the D1 API to execute queries and proxy stuff through it, there's some community stuff to like a JDBC driver using the API: https://github.com/isaac-mcfadyen/d1-jdbc-driver
Works with Datagrip and probably anything that works with jdbc. Looks like dbeaver uses odbc, I assume you could make your own driver wrapping the API, that's all driver/db client stuff though

niconiahi•7/23/24, 6:09 PM

I didn't know the don't ha ve a remote protocol. Thanks!

what a pity

I'll check that out and see how hard the wrapper is

Max (@rozenmd)OP•7/24/24, 10:03 AM

read replication hasn't shipped yet

Chinoman10•7/24/24, 12:22 PM

I just want to "show off" this project I found, which I've requested the devs for support for CF D1 and they added it a few days ago. I just tested it now and it works really well.
I prefer this interface over the previously shared "D1 Manager" (that is also only limited to D1s), and they are both open-source and self-hostable.
https://github.com/invisal/libsql-studio/

GitHub

GitHub - invisal/libsql-studio: A lightweight SQLite graphical clie...

A lightweight SQLite graphical client on your browser. It also support connecting to LibSQL/Turso/rqlite - invisal/libsql-studio

CChinoman10 I just want to "show off" this project I found, which I've requested the devs fo...

Talaat•7/24/24, 12:30 PM

i was looking for something similar
Thanks for sharing

KWAC•7/25/24, 1:42 PM

Is anyone here using nuxt with D1?

zensin•7/26/24, 12:56 AM

you can use D1 remotely right? doesn't have to be in workers, does it?

zensin•7/26/24, 1:01 AM

nvm. It apparently doesn't support remote connections outside cloudflare. Someone must have tried to make a worker for this purpose though

zensin•7/26/24, 1:02 AM

https://github.com/WordPress/sqlite-database-integration/pull/9

GitHub

SQLite support via a MySQL parser by adamziel · Pull Request #9 · W...

Description
Implements SQLite support using a MySQL lexer instead of regular expressions to:

Greatly improve security
Solve most WordPress core unit test failures
Make the plugin more maintenable
...

Zzensin nvm. It apparently doesn't support remote connections outside cloudflare. Someon...

Chaika•7/26/24, 1:03 AM

well I mean it's Sqlite so there's no native remote connection format, but you can send queries via the Cloudflare API (but they're all funneled through one of the cores right now, so user -> api in core -> d1 running somewhere) or make a Worker for it which run everywhere (but you'd have to set it up/secure it and pay for each invocation)

CChaika well I mean it's Sqlite so there's no native remote connection format, but you c...

zensin•7/26/24, 1:05 AM

you can send queries via the Cloudflare API

you mean via the implementation already in D1 client library?

they're all funneled through one of the cores

What is meant by this?

Zzensin > you can send queries via the Cloudflare API you mean via the implementation a...

Chaika•7/26/24, 1:07 AM

you mean via the implementation already in D1 client library?

I mean you can send a query via the Cloudflare API to be executed
https://developers.cloudflare.com/api/operations/cloudflare-d1-raw-database-query?schema_url=https%3A%2F%2Fraw.githubusercontent.com%2Fcloudflare%2Fapi-schemas%2Fmain%2Fopenapi.yaml

What is meant by this?

Worker goes
User -> Cloudflare data center (hopefully pretty close) -> D1 Instance (running somewhere)
API Queries go
User -> Cloudflare data center -> Cloudflare core (either us or eu, I think the D1 API is just US/west coast but not 100% sure) -> D1 Instance (running somewhere) and back. Of course very big difference espec if your D1 instance was in Oceanic region/Australia for example, lots of going back and forth

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Zzensin https://github.com/WordPress/sqlite-database-integration/pull/9

Chaika•7/26/24, 1:08 AM

this is just for local sqlite as far as I can see but you could probably use some of the base work and hook it up to a worker/the api, would just have to handle responses/latency/etc, not sure how well Wordpress would play with it, espec with some of those plugins which make a crazy amount of queries, when D1 isn't same machine/can have decent query times

Isaac McFadyen•7/26/24, 1:10 AM

I'd be doubtful that you'd be able to hook that Wordpress SQLite update up to D1 in an efficient/effective way. It's already a bit tenous last I tried it because it's translating the MySQL commands to SQLite, adding an extra layer in there with a bunch of latency will only make things worse.

CChaika this is just for local sqlite as far as I can see but you could probably use som...

zensin•7/26/24, 1:10 AM

I wasn't thinking of only wordpress, though one does occasionally get forced to deal with it. I mentioned WP bc they have an effort underway (I'd say it's in beta/RC stage) to adapt all their mysql queries to sqlite. I was thinking about maybe porting their mysql lexer to a cloudflare worker.

zensin•7/26/24, 1:10 AM

that is if the lexer plays nice with D1

IIsaac McFadyen I'd be doubtful that you'd be able to hook that Wordpress SQLite update up to D1...

Chaika•7/26/24, 1:10 AM

start batching all of the queries or something lol

Chaika•7/26/24, 1:11 AM

yea would probably be a ton of work to try to get it semi-usable

Zzensin that is if the lexer plays nice with D1

Chaika•7/26/24, 1:12 AM

well D1 is just SQlite so I would assume it'd be fine if they got it working with sqlite there, there is some additional security restrictions (like with query/etc anti-abuse limits) they use that have cropped up in the past but I think they've eased up on a lot of them

Isaac McFadyen•7/26/24, 1:13 AM

There's also a few commands that are restricted in D1 vs regular SQLite for security reasons (some of the PRAGMAs) but yeah, otherwise it's pretty similar.

zensin•7/26/24, 1:13 AM

then probably most annoying points would be:

porting from PHP (Cause who wants to run php in a worker, though you can transpile)
transport of mysql connection over websockets (Idk if that's natively supported/option)

Zzensin then probably most annoying points would be: - porting from PHP (Cause who wants...

Isaac McFadyen•7/26/24, 1:14 AM

D1 can't connect via the MySQL protocol at all, so the second part wouldn't really be a consideration - you'd need to just adapt the commands to be proper SQLite queries before sending to D1.

Isaac McFadyen•7/26/24, 1:15 AM

If you mean connecting to actual MySQL from Workers then I believe there were some efforts to get some of the MySQL drivers to work with the Workers TCP Sockets API.

Isaac McFadyen•7/26/24, 1:15 AM

Not sure how those ended up.

IIsaac McFadyen D1 can't connect via the MySQL protocol at all, so the second part wouldn't real...

zensin•7/26/24, 1:15 AM

right, that's what the worker would be for. Translating the mysql connection to sqlite commands

Chaika•7/26/24, 1:16 AM

well workers can't do inbound tcp and websocket stuff would be inefficient

also https://github.com/nora-soderlund/cloudflare-dynamic-bindings might be of interest

Similar Threads