hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in the future? I am not a

hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in the future? I am not asking for any timelines, as I don't have any need right now, but I want to understand if I am locking myself into a tech choice that I may not be able to work with later. (I know that I can create multiple databases, but I fear the spreading my data over multiple databses will make things way too complicated for me)

RRick hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in th...

user•11/20/23, 2:10 PM

^^ this

Uuser https://user.you-sk.id/plCW7Cg4qgki Is there any way to get more than 2gb?

Max (@rozenmd)•11/20/23, 2:25 PM

today? no, it's a hard limit we're working on increasing

MMax (@rozenmd)today? no, it's a hard limit we're working on increasing

user•11/20/23, 3:04 PM

Is there any sort of guess on when the hard limit might be increased?

user•11/20/23, 3:04 PM

I read into it and understand that it’s a difficult task to maintain for CF

user•11/20/23, 3:04 PM

Just curious if there’s any guesses about progress

Uuser Is there any sort of guess on when the hard limit might be increased?

Max (@rozenmd)•11/20/23, 3:15 PM

we don't guess/provide estimates - once we increase the limit there will likely be a large announcement about it

RRick hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in th...

Matt Silverlock•11/20/23, 3:17 PM

Not in the next 6+ months (at least). Hyperdrive + a traditional RDBMS is likely going to be a better fit if you truly need 50GB in a single database (or have an existing DB you want to migrate) for the time being. Different use-cases.

MMax (@rozenmd)we don't guess/provide estimates - once we increase the limit there will likely ...

user•11/20/23, 3:21 PM

Alright, I assume the D1 role will be pongtsted?

clearwater4227•11/21/23, 2:07 AM

Hello all, I was looking to build with D1 but dont see a way to provision tables/schema through terraform. I just see creating databases... Am I missing something?

Cclearwater4227 Hello all, I was looking to build with D1 but dont see a way to provision table...

Cyb3r-Jak3•11/21/23, 2:33 AM

You typically wouldn't create the schema through terraform and better to use something like Drizzle (https://orm.drizzle.team/docs/quick-sqlite/d1) to manage it

Cloudflare D1 - Drizzle ORM

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

clearwater4227•11/21/23, 12:04 PM

ok, interesting....so this looks like create/define schema in js and it will create schema/tables if they dont exist?

Cclearwater4227 Hello all, I was looking to build with D1 but dont see a way to provision table...

Max (@rozenmd)•11/21/23, 12:42 PM

Generally terraform is used to spin up infrastructure, I haven't heard of folks using it for SQL commands

clearwater4227•11/21/23, 1:03 PM

I use terraform for building out my bigquery database tables. I was going to move some of my tables to D1 for performance and cost reasons so I was looking at integrating that into my terraform setup.

clearwater4227•11/21/23, 1:05 PM

kind of build tables and workers for a few tables, setup terraform setup/deploy, migrate things over, and press go. Terraform is nice for schema management as well, at least for bigquery, as I can add and change columns and the bigquery terraform integration handles the changes and migrations.

Alexey•11/21/23, 4:16 PM

hey folks. I'm new to D1 (and databases in General).
My current problem is data conversion. It seems to me I need to manually convert JS data types to SQL data types and vice versa.
For example, for D1 BOOLEAN values I get 00/11 in my function code. I expect to have truetrue/

false

false

in my function code instead.
I looked into some ORMs but didn't find that functionality for automatic types conversion.

What are the ways to handle this problem? How do you approach it?

AAlexey hey folks. I'm new to D1 (and databases in General). My current problem is data ...

Bram•11/21/23, 4:30 PM

SQLite does not have a boolean storage class, but is instead stored as 0 or 1. You can use an ORM for this, I recommend

drizzle

drizzle

, a relatively new ORM that has a lot of momentum and is build with edge in mind.

Bram•11/21/23, 4:34 PM

it let's you declare sql schema's directly in typescript, where you can hint the ORM that that the integer should be interpreted as a boolean on application level: https://orm.drizzle.team/docs/column-types/sqlite integer('id', { mode: 'boolean' })integer('id', { mode: 'boolean' }), same for timestamps, json etc

SQLite column types - Drizzle ORM

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

Alexey•11/21/23, 4:35 PM

ok, interesting, will give drizzle a try. Thank for the suggestion @Bram

Unsmart•11/21/23, 5:32 PM

The main way would be to create a worker as a proxy

James•11/21/23, 5:53 PM

I think this is that same obscure issue we've seen a few times, cc @Matt: https://community.cloudflare.com/t/d1-database-is-crashed/583564

Cloudflare Community

D1 database is crashed

My D1 database is suddenly crashed. And it is not recovered. In D1 tab, there is a message “Failed to reach database. Please try again later”. What’s going on?

Cyb3r-Jak3•11/21/23, 6:49 PM

Old vs new backend. My guess is the first one hasn’t had the backend migrated.

RRed Spider Is it possible to bulk insert via the API? I have an array of items i want to in...

Hello, I’m Allie!•11/21/23, 7:53 PM

Depending on the size of the items, and whether they are rows in a single table, or multiple tables, you might be able to do a multi-insert

Hello, I’m Allie!•11/21/23, 7:54 PM

INSERT INTO table1 (column1,column2 ,..)
VALUES 
   (value1,value2 ,...),
   (value1,value2 ,...),
    ...
   (value1,value2 ,...);

INSERT INTO table1 (column1,column2 ,..)
VALUES 
   (value1,value2 ,...),
   (value1,value2 ,...),
    ...
   (value1,value2 ,...);

Hello, I’m Allie!•11/21/23, 7:54 PM

Something like this

EXtremeExploit•11/21/23, 8:17 PM

is D1 down?

EXtremeExploit•11/21/23, 8:18 PM

alrighty just wanted to make sure it wasnt only me, thankie

Unsmart•11/21/23, 8:18 PM

Seems fine for me... is this every query or just some?

EXtremeExploit•11/21/23, 8:18 PM

all of them

EXtremeExploit•11/21/23, 8:18 PM

even the site fails to fetch the data

EXtremeExploit•11/21/23, 8:22 PM

EXtremeExploit•11/21/23, 8:23 PM

Unsmart•11/21/23, 8:24 PM

Looks like 1 of my databases isnt loading in the dashboard for some reason not sure whats going on with it but I havent used it at all. My other 2 databases seem just fine

Matt Silverlock•11/21/23, 8:33 PM

Account IDs and database IDs please, folks

mceu•11/21/23, 8:35 PM

Is anyone using d1 with Superset?

1.) Understand Your Tools:
Apache Superset: It's an open-source data exploration and visualization platform.
Cloudflare D1: A distributed SQL database that is part of Cloudflare’s suite of products. It's designed to be easily scalable and managed in a cloud environment.

2.) Prerequisites:
Ensure you have Apache Superset installed and properly configured.
Have access to a Cloudflare D1 instance and ensure it's operational.

3.) Database Connectivity:
Superset connects to various data sources through SQL Alchemy, a Python SQL toolkit and Object-Relational Mapping (ORM) library.
You need to find out if Cloudflare D1 provides a SQL Alchemy-compatible connector. If it does, you can use it to establish a connection between Superset and D1.
If there's no direct connector available, you might need to use a generic SQL Alchemy connector or a connector for a database that's compatible with Cloudflare D1's SQL dialect.

4.) Configure the Database Connection:
In Superset, go to the ‘Data’ menu and select ‘Databases’.
Add a new database connection.
Enter the connection details for your Cloudflare D1 database. This typically includes the database type, connection URL (which comprises the username, password, server address, and database name), and other parameters.

5.) Test the Connection:
After configuring, test the connection to ensure that Superset can communicate with Cloudflare D1.
Address any connection errors by reviewing the connection parameters and consulting the documentation for both Superset and Cloudflare D1.

6.) Import and Visualize Data:
Once connected, you can start importing data from Cloudflare D1 into Superset.
Use Superset’s features to create dashboards, charts, and reports based on the data stored in your Cloudflare D1 database.

MMatt Silverlock Account IDs and database IDs please, folks 🙂

Unsmart•11/21/23, 8:35 PM

Account: dc941e8156f4a1336ca08481cb6d4222dc941e8156f4a1336ca08481cb6d4222
Database: a42194f9-3137-41a1-8f64-df0cf045a6d6a42194f9-3137-41a1-8f64-df0cf045a6d6

mceu•11/21/23, 8:36 PM

Per the Superset folks - "Hi there, I can’t say I’ve heard of such an integration… but if there’s a sqlalchemy dialect for it (which I don’t see at a cursory glance) then it could be added. It might be worth reaching out to the D1 and/or SQLAlchemy communities to see if they’ve been connected. If so, we can probably carry that into Superset :)"

MMatt Silverlock Account IDs and database IDs please, folks 🙂

EXtremeExploit•11/21/23, 8:37 PM

account id (from workers overview): 6099b559aa0e86a9471d30f3dccafec66099b559aa0e86a9471d30f3dccafec6
db id: 2452c369-1d24-4bc9-9994-029c71619ca32452c369-1d24-4bc9-9994-029c71619ca3

EEXtremeExploit account id (from workers overview): `6099b559aa0e86a9471d30f3dccafec6` db id: `2...

Matt Silverlock•11/21/23, 9:05 PM

@EXtremeExploit @Unsmart | Tech debt We see a small number of databases failing to start - coinciding with a release. We're about to roll that back.

EXtremeExploit•11/21/23, 9:06 PM

alrighty thankie

MMatt Silverlock Account IDs and database IDs please, folks 🙂

oscar•11/21/23, 9:10 PM

Hey thanks!
account id: 34c0d8412965b33f782ad043d255542c
database id: 5d736d0e-4b0f-4726-935e-5ec1d1352825
Shows up as reachable and can see the data, but my worker still fails with D1_ERROR

Ooscar Hey thanks! account id: 34c0d8412965b33f782ad043d255542c database id: 5d736d0e-4...

Matt Silverlock•11/21/23, 9:16 PM

If you can see the data (in wrangler? the dashboard?) then you may have another issue. D1_ERRORD1_ERROR may be an issue with your query.

oscar•11/21/23, 9:22 PM

oh, sorry! I didn't realize D1_ERROR was for queries too, I thought I had seen a different message earlier

clearwater4227•11/21/23, 9:46 PM

hey all, I am trying to create a D1 database via terraform and am getting this error:

Error: failed to create D1 database
with module.lookup_codes.cloudflare_d1_database.lookup-codes_database
on .terraform/modules/lookup_codes/cloudflare.tf line 29, in resource "cloudflare_d1_database" "lookup-codes_database":
resource "cloudflare_d1_database" "lookup-codes_database" {
error from makeRequest: Authentication error (10000)

It seems pretty easy to do...not sure what would be wrong. Was able to create from UI. Here is what Im doing:

resource "cloudflare_d1_database" "lookup-codes_database" {
account_id = var.cloudflare_account_id
name = "lookup_codes_database"
}

This account id is right as it create other resources just fine...

hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in the future? I am not a

Similar Threads