Account IDs and database IDs please, folks 🙂

Account IDs and database IDs please, folks

mceu•11/21/23, 8:35 PM

Is anyone using d1 with Superset?

1.) Understand Your Tools:
Apache Superset: It's an open-source data exploration and visualization platform.
Cloudflare D1: A distributed SQL database that is part of Cloudflare’s suite of products. It's designed to be easily scalable and managed in a cloud environment.

2.) Prerequisites:
Ensure you have Apache Superset installed and properly configured.
Have access to a Cloudflare D1 instance and ensure it's operational.

3.) Database Connectivity:
Superset connects to various data sources through SQL Alchemy, a Python SQL toolkit and Object-Relational Mapping (ORM) library.
You need to find out if Cloudflare D1 provides a SQL Alchemy-compatible connector. If it does, you can use it to establish a connection between Superset and D1.
If there's no direct connector available, you might need to use a generic SQL Alchemy connector or a connector for a database that's compatible with Cloudflare D1's SQL dialect.

4.) Configure the Database Connection:
In Superset, go to the ‘Data’ menu and select ‘Databases’.
Add a new database connection.
Enter the connection details for your Cloudflare D1 database. This typically includes the database type, connection URL (which comprises the username, password, server address, and database name), and other parameters.

5.) Test the Connection:
After configuring, test the connection to ensure that Superset can communicate with Cloudflare D1.
Address any connection errors by reviewing the connection parameters and consulting the documentation for both Superset and Cloudflare D1.

6.) Import and Visualize Data:
Once connected, you can start importing data from Cloudflare D1 into Superset.
Use Superset’s features to create dashboards, charts, and reports based on the data stored in your Cloudflare D1 database.

MMatt Silverlock Account IDs and database IDs please, folks 🙂

Unsmart•11/21/23, 8:35 PM

Account:

dc941e8156f4a1336ca08481cb6d4222

dc941e8156f4a1336ca08481cb6d4222

Database:

a42194f9-3137-41a1-8f64-df0cf045a6d6

a42194f9-3137-41a1-8f64-df0cf045a6d6

mceu•11/21/23, 8:36 PM

Per the Superset folks - "Hi there, I can’t say I’ve heard of such an integration… but if there’s a sqlalchemy dialect for it (which I don’t see at a cursory glance) then it could be added. It might be worth reaching out to the D1 and/or SQLAlchemy communities to see if they’ve been connected. If so, we can probably carry that into Superset :)"

MMatt Silverlock Account IDs and database IDs please, folks 🙂

EXtremeExploit•11/21/23, 8:37 PM

account id (from workers overview):

6099b559aa0e86a9471d30f3dccafec6

6099b559aa0e86a9471d30f3dccafec6

db id:

2452c369-1d24-4bc9-9994-029c71619ca3

2452c369-1d24-4bc9-9994-029c71619ca3

EEXtremeExploit account id (from workers overview): `6099b559aa0e86a9471d30f3dccafec6` db id: `2...

Matt SilverlockOP•11/21/23, 9:05 PM

@EXtremeExploit @Unsmart | Tech debt We see a small number of databases failing to start - coinciding with a release. We're about to roll that back.

EXtremeExploit•11/21/23, 9:06 PM

alrighty thankie

MMatt Silverlock Account IDs and database IDs please, folks 🙂

oscar•11/21/23, 9:10 PM

Hey thanks!
account id: 34c0d8412965b33f782ad043d255542c
database id: 5d736d0e-4b0f-4726-935e-5ec1d1352825
Shows up as reachable and can see the data, but my worker still fails with D1_ERROR

Ooscar Hey thanks! account id: 34c0d8412965b33f782ad043d255542c database id: 5d736d0e-4...

Matt SilverlockOP•11/21/23, 9:16 PM

If you can see the data (in wrangler? the dashboard?) then you may have another issue.

D1_ERROR

D1_ERROR

may be an issue with your query.

oscar•11/21/23, 9:22 PM

oh, sorry! I didn't realize D1_ERROR was for queries too, I thought I had seen a different message earlier

clearwater4227•11/21/23, 9:46 PM

hey all, I am trying to create a D1 database via terraform and am getting this error:

Error: failed to create D1 database
with module.lookup_codes.cloudflare_d1_database.lookup-codes_database
on .terraform/modules/lookup_codes/cloudflare.tf line 29, in resource "cloudflare_d1_database" "lookup-codes_database":
resource "cloudflare_d1_database" "lookup-codes_database" {
error from makeRequest: Authentication error (10000)

It seems pretty easy to do...not sure what would be wrong. Was able to create from UI. Here is what Im doing:

resource "cloudflare_d1_database" "lookup-codes_database" {
account_id = var.cloudflare_account_id
name = "lookup_codes_database"
}

This account id is right as it create other resources just fine...

clearwater4227•11/21/23, 9:46 PM

wait...I think I know...

MMatt Silverlock @EXtremeExploit @Unsmart | Tech debt We see a small number of databases failing ...

Matt SilverlockOP•11/21/23, 9:48 PM

@EXtremeExploit @Unsmart | Tech debt we’re seeing databases recover (aka successfully load again).

EXtremeExploit•11/21/23, 9:48 PM

YokoWasis•11/22/23, 12:52 AM

is D1 is down right now ?

YYokoWasis is D1 is down right now ?

Cyb3r-Jak3•11/22/23, 12:56 AM

Shouldn't be. What errors are you getting?

YokoWasis•11/22/23, 12:56 AM

YokoWasis•11/22/23, 12:58 AM

i can't query my database

YYokoWasis i can't query my database

Cyb3r-Jak3•11/22/23, 12:58 AM

Can you share your account ID and database ID?

YokoWasis•11/22/23, 12:58 AM

CCyb3r-Jak3 Can you share your account ID and database ID?

YokoWasis•11/22/23, 12:58 AM

id : df608b9e06cca37a7fc399c8ca86eab4
database : 0f3e37af-ecd2-4e73-9766-72e181305f10

YYokoWasis id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e1813...

Cyb3r-Jak3•11/22/23, 12:59 AM

CC @Matt

YokoWasis•11/22/23, 1:03 AM

anyone experiencing the same, or is it just me ?

Cyb3r-Jak3•11/22/23, 1:09 AM

There were some reports earlier (d1-database) but they rolled back

Unsmart•11/22/23, 1:17 AM

My table that was broken seems like it still is so

maybe its not fully rolled back yet or they redid it not sure. Not too worried as I wouldnt use anything production with a beta product and it happened on a table I dont even use and I dont know why I made KEKW

YokoWasis•11/22/23, 1:23 AM

anyone can give an eta about this ?

Unsmart•11/22/23, 1:24 AM

It's a beta product so probably whenever they are online again tomorrow it will be looked at

YokoWasis•11/22/23, 1:31 AM

YYokoWasis id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e1813...

Matt SilverlockOP•11/22/23, 1:46 AM

We’re investigating but can’t promise this will be resolved immediately

UUnsmart My table that was broken seems like it still is so 🤷 maybe its not fully rolled...

Matt SilverlockOP•11/22/23, 1:46 AM

Fully rolled back. A single table being broken is something else. The issue prevented the DB from loading. There is no such thing as a “partial” load - we don’t load incremental pages. I would not associate with today’s issue.

Unsmart•11/22/23, 1:48 AM

I meant database not table in that message... I dont think it even has any tables but not really sure since I dont use that database at all and dont remember making it

Unsmart•11/22/23, 1:51 AM

as in I still get this and it hasnt changed since the rollback (specifically the last database):

YYokoWasis id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e1813...

Matt SilverlockOP•11/22/23, 2:17 AM

We see this database working now.

MMatt Silverlock We see this database working now.

YokoWasis•11/22/23, 2:35 AM

thanks, it's working now

YokoWasis•11/22/23, 2:37 AM

is this happened for all of d1, or specific few databases ? should I create a database as a mirror in case this happened again ?

YYokoWasis is this happened for all of d1, or specific few databases ? should I create a da...

Matt SilverlockOP•11/22/23, 2:40 AM

We saw < 10 databases impacted here.

James•11/22/23, 2:54 AM

It's not the first time folks have hit this same error message before. I'm not sure if the root cause is the same or not, but especially for issues like this that externally seem to be recurring ones, it would be really appreciated if a status page could be put up in future. It saves a lot of the reactive back and forth that happens here, and only improves the transparency we all know and love Cloudflare for MeowHeartCloudflare

MMatt Silverlock We saw < 10 databases impacted here.

YokoWasis•11/22/23, 2:59 AM

thanks, if this is the case I will create a mirror for my database as a backup.

YokoWasis•11/22/23, 3:02 AM

how to download full backup of database ?

James•11/22/23, 3:53 AM

Retro status page went up: https://www.cloudflarestatus.com/incidents/2j7wtjt39ypl

Small number of D1 databases failed to execute queries

James•11/22/23, 3:53 AM

Thanks all MeowHeartCloudflare

MMatt Silverlock We see this database working now.

YokoWasis•11/22/23, 4:22 AM

it's down again

YokoWasis•11/22/23, 5:10 AM

anyone knows how to do full database backup with wrangler ?

YYokoWasis anyone knows how to do full database backup with wrangler ?

James•11/22/23, 5:12 AM

You can’t yet directly with wrangler, but you can use this great community solution: https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

federo•11/22/23, 11:39 AM

can anyone help me understand how is this viable if a database can only hold 2gb of data?

zegevlier•11/22/23, 11:42 AM

(not a team member, just from my knowledge as an outsider)
First: The goal is still to increase this during the beta. Second: With D1, you can have a lot of seperate databases, all with a limit of 2gb. If your application can support sharding the DB like this, there isn't really a limit to the amount of data you can store. To me it seems perfect for a lot of SaaS applications, where each customer, or even each project, would have their own database.

federo•11/22/23, 11:44 AM

so if a client would then need to make a query, the query would have to first look at which database to look into and perhaps query the data across databases?

Ffedero so if a client would then need to make a query, the query would have to first lo...

Max (@rozenmd)•11/22/23, 11:46 AM

a certain issue manager that many folks dislike handles multi-tenancy that way, yeah - one db per customer.

you keep a highly available store mapping something (customerId, a subdomain, etc) to the database ID, then pass the databaseID to your client (we're working on making the d1-api support this nicely)

Max (@rozenmd)•11/22/23, 11:48 AM

migrations are also a pain atm with this model: https://github.com/cloudflare/workers-sdk/issues/4471

federo•11/22/23, 11:48 AM

but if a customers needs way more than 2gb of data, then do you see this viable?
I am trying to setup a flow where a js script on websites sends events data to a cloudflare worker, from which I am trying to understand how to best forward it to a database

Max (@rozenmd)•11/22/23, 11:49 AM

for events data d1 is going to have a bad time, you might want to look into https://developers.cloudflare.com/analytics/analytics-engine/

Account IDs and database IDs please, folks 🙂

Similar Threads