If i have an cloud app that has many customers, what is the Cloudflare architecture recommendation f

If i have an cloud app that has many customers, what is the Cloudflare architecture recommendation for stateful resources (like D1 and Queues)?
Is it better to stick with simple multi-tenancy (option-A), or is it better (or even possible) to use the Cloudflare API's to dynamically provision for each customer (option-B).
In the later case each customer would have their own set of D1 databases and Queues.
I don't see any doc or sample code demonstrating option-B?

Mmarko.333 If i have an cloud app that has many customers, what is the Cloudflare architect...

marko.333OP•7/19/24, 10:19 PM

Opps, appears a similar question was literally asked today!-)
It sounds like the answer to my question is that "option-B" is not currently supported?
I had the impression it may be, since there are API's that appear intended to dynamically provision resources?
https://developers.cloudflare.com/api/operations/cloudflare-d1-create-database

Mmarko.333 Opps, appears a similar question was literally asked today!-) It sounds like ...

Isaac McFadyen•7/20/24, 1:36 AM

So there are APIs to dynamically provision resources, but you can't bind them to use from a Worker without a redeploy of that Worker.

Isaac McFadyen•7/20/24, 1:36 AM

The "binding dynamically to the Worker" is what's on the roadmap/being worked on.

Isaac McFadyen•7/20/24, 1:36 AM

You can manually query D1 via the Cloudflare API but it's much slower than binding directly to a Worker and has very strict rate-limits (1200 calls per 5 minutes I believe?)

IIsaac McFadyen The "binding dynamically to the Worker" is what's on the roadmap/being worked on...

marko.333OP•7/20/24, 1:52 AM

Gotcha, thx. I got ahead of myself when Claude 3.5 gave me this hallucination...

db = env.__D1_BETA__D1DatabaseFactory(databaseId);

db = env.__D1_BETA__D1DatabaseFactory(databaseId);

Is there a formal Issue publicly accessible to track this work?
thx

BBen Hello I have a question about future plans. Currently in the D1 limits page it s...

Max (@rozenmd)•7/20/24, 11:14 AM

actively building it (during work hours anyway)

Max (@rozenmd)•7/20/24, 11:16 AM

also https://github.com/nora-soderlund/cloudflare-dynamic-bindings might be of interest

MMax (@rozenmd)also https://github.com/nora-soderlund/cloudflare-dynamic-bindings might be of i...

marko.333OP•7/20/24, 11:47 AM

i'm not sure i would call that github project "dynamic" (maybe "cloudflare-gen-bindings")? Regardless - cool hack.
I wish there was an open discussions page to discuss requests and future direction?
Besides being able to dynamically create resources, it would also be nice to build some higher-level abstractions (at least for common usecases), to help simplify the management.
For example, it would be great if there was some form of higher-level "sharding" support that only required a resource to be configured with a "sharding key" (Example: A D1 "foobar" db shared by "customer_id"). That resource (D1, or Queue, or ...) would then require the shard-key to operate on it.
Under the covers Cloudflare would then manage "foobar" by appropriately creating a D1 instance when a new key is referenced, or finding/binding the instance when it does exist, and Migrations could also be done across the entire "foobar" cluster?

Mmarko.333 i'm not sure i would call that github project "dynamic" (maybe "cloudflare-gen-b...

Max (@rozenmd)•7/20/24, 11:53 AM

that's roughly how we're designing d1 dynamic bindings

Max (@rozenmd)•7/20/24, 11:58 AM

the migrations story becomes "fun" with that though, since you can now make potentially millions of DBs, we don't want migrations to simultaneously wake up millions of DBs

MMax (@rozenmd)that's roughly how we're designing d1 dynamic bindings

marko.333OP•7/20/24, 1:19 PM

OK, great to know that more coolness is on the way.
To add another dimension of complexity - hopefully there is also some form of integration into "Gradual Deployments"?
https://developers.cloudflare.com/workers/configuration/versions-and-deployments/gradual-deployments/
Maybe that could be part of the story to ease the cost of migrations?
Btw, is there similar work in-progress to support dynamic binding (and sharding) for Queues?
Thx Max.

Cory•7/20/24, 8:51 PM

ARe there any plans to be able to subscribe to updates?

JoeDoe•7/21/24, 6:30 AM

Is there some client library or something that I can use to access D1 database remotely, but still use the local one for testing without having to write double queries for the remote and local one?

mohitgangrade•7/21/24, 7:09 AM

[ERROR] not authorized: SQLITE_AUTH

I've even tried creating a custom token and uploading the SQL file using the REST API. I keep getting this error after the file gets done uploading.

mohitgangrade•7/21/24, 7:09 AM

mohitgangrade•7/21/24, 7:09 AM

I get this error even in the Cloudflare Dashboard

kaushal•7/21/24, 12:24 PM

hello

kaushal•7/21/24, 12:24 PM

anybody here ?

Flare•7/21/24, 12:32 PM

https://dontasktoask.com/

Don't ask to ask, just ask

Gerbuuun•7/21/24, 8:50 PM

You could query the sqlite_sequencesqlite_sequence table but like Leo said you can get race conditions

Zeeshan•7/22/24, 5:09 AM

Hi folks. i have a bunch of data in a csv file which i would like to dump into my table i d1 db. Which is the best way to do this? Seems like d1 doesn't support csv yet (cmiiw)

Original message was deleted

JoeDoe•7/22/24, 7:40 AM

what dat

Original message was deleted

JoeDoe•7/22/24, 7:42 AM

this can connect to D1 database remotely?

Phaze•7/23/24, 1:45 PM

Heyo, I'm trying to select a map by an id but some of the map ids are all numbers, and something keeps trying to format the id, that is a string, into a float. I can't work out if this is a D1 issue or an issue with workers-rs

https://cdn.phaz.uk/i/Code_5UINmeAWon.png https://cdn.phaz.uk/i/Code_wAffgOTnIe.png

Help would be greatly appreciated :)

PPhaze Heyo, I'm trying to select a map by an id but some of the map ids are all number...

Phaze•7/23/24, 5:22 PM

I'm still getting this issue, even with other parts of my code, it's set to string everywhere yet it's still trying to convert it to a float

niconiahi•7/23/24, 5:43 PM

are there any plans on exposing a connection string to control the product d1 database? I use DBeaver and far far more superior than the db visualizer that you guys have

has there been any conversations around this? it's kind of an obvious move IMO as there is so much tooling already to be used that is better to diagnose a db

it takes 5 more times to do whatever i have to do on the d1 db viewer than DBeaver

Nniconiahi are there any plans on exposing a connection string to control the product d1 da...

Chaika•7/23/24, 5:51 PM

what would that connection string be of? D1 is SQlite, there is no remote sqlite protocol

Chaika•7/23/24, 5:52 PM

You can use the D1 API to execute queries and proxy stuff through it, there's some community stuff to like a JDBC driver using the API: https://github.com/isaac-mcfadyen/d1-jdbc-driver
Works with Datagrip and probably anything that works with jdbc. Looks like dbeaver uses odbc, I assume you could make your own driver wrapping the API, that's all driver/db client stuff though

niconiahi•7/23/24, 6:09 PM

I didn't know the don't ha ve a remote protocol. Thanks!

what a pity

I'll check that out and see how hard the wrapper is

Max (@rozenmd)•7/24/24, 10:03 AM

read replication hasn't shipped yet

Chinoman10•7/24/24, 12:22 PM

I just want to "show off" this project I found, which I've requested the devs for support for CF D1 and they added it a few days ago. I just tested it now and it works really well.
I prefer this interface over the previously shared "D1 Manager" (that is also only limited to D1s), and they are both open-source and self-hostable.
https://github.com/invisal/libsql-studio/

GitHub

GitHub - invisal/libsql-studio: A lightweight SQLite graphical clie...

A lightweight SQLite graphical client on your browser. It also support connecting to LibSQL/Turso/rqlite - invisal/libsql-studio

CChinoman10 I just want to "show off" this project I found, which I've requested the devs fo...

Talaat•7/24/24, 12:30 PM

i was looking for something similar
Thanks for sharing

KWAC•7/25/24, 1:42 PM

Is anyone here using nuxt with D1?

zensin•7/26/24, 12:56 AM

you can use D1 remotely right? doesn't have to be in workers, does it?

zensin•7/26/24, 1:01 AM

nvm. It apparently doesn't support remote connections outside cloudflare. Someone must have tried to make a worker for this purpose though

zensin•7/26/24, 1:02 AM

https://github.com/WordPress/sqlite-database-integration/pull/9

GitHub

SQLite support via a MySQL parser by adamziel · Pull Request #9 · W...

Description
Implements SQLite support using a MySQL lexer instead of regular expressions to:

Greatly improve security
Solve most WordPress core unit test failures
Make the plugin more maintenable
...

Zzensin nvm. It apparently doesn't support remote connections outside cloudflare. Someon...

Chaika•7/26/24, 1:03 AM

well I mean it's Sqlite so there's no native remote connection format, but you can send queries via the Cloudflare API (but they're all funneled through one of the cores right now, so user -> api in core -> d1 running somewhere) or make a Worker for it which run everywhere (but you'd have to set it up/secure it and pay for each invocation)

CChaika well I mean it's Sqlite so there's no native remote connection format, but you c...

zensin•7/26/24, 1:05 AM

you can send queries via the Cloudflare API

you mean via the implementation already in D1 client library?

they're all funneled through one of the cores

What is meant by this?

Zzensin > you can send queries via the Cloudflare API you mean via the implementation a...

Chaika•7/26/24, 1:07 AM

you mean via the implementation already in D1 client library?

I mean you can send a query via the Cloudflare API to be executed
https://developers.cloudflare.com/api/operations/cloudflare-d1-raw-database-query?schema_url=https%3A%2F%2Fraw.githubusercontent.com%2Fcloudflare%2Fapi-schemas%2Fmain%2Fopenapi.yaml

What is meant by this?

Worker goes
User -> Cloudflare data center (hopefully pretty close) -> D1 Instance (running somewhere)
API Queries go
User -> Cloudflare data center -> Cloudflare core (either us or eu, I think the D1 API is just US/west coast but not 100% sure) -> D1 Instance (running somewhere) and back. Of course very big difference espec if your D1 instance was in Oceanic region/Australia for example, lots of going back and forth

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Zzensin https://github.com/WordPress/sqlite-database-integration/pull/9

Chaika•7/26/24, 1:08 AM

this is just for local sqlite as far as I can see but you could probably use some of the base work and hook it up to a worker/the api, would just have to handle responses/latency/etc, not sure how well Wordpress would play with it, espec with some of those plugins which make a crazy amount of queries, when D1 isn't same machine/can have decent query times

Isaac McFadyen•7/26/24, 1:10 AM

I'd be doubtful that you'd be able to hook that Wordpress SQLite update up to D1 in an efficient/effective way. It's already a bit tenous last I tried it because it's translating the MySQL commands to SQLite, adding an extra layer in there with a bunch of latency will only make things worse.

CChaika this is just for local sqlite as far as I can see but you could probably use som...

zensin•7/26/24, 1:10 AM

I wasn't thinking of only wordpress, though one does occasionally get forced to deal with it. I mentioned WP bc they have an effort underway (I'd say it's in beta/RC stage) to adapt all their mysql queries to sqlite. I was thinking about maybe porting their mysql lexer to a cloudflare worker.

zensin•7/26/24, 1:10 AM

that is if the lexer plays nice with D1

IIsaac McFadyen I'd be doubtful that you'd be able to hook that Wordpress SQLite update up to D1...

Chaika•7/26/24, 1:10 AM

start batching all of the queries or something lol

If i have an cloud app that has many customers, what is the Cloudflare architecture recommendation f

Similar Threads

Similar Threads

Similar Threads