hello, most probably you are already tracking this; do you have an issue or place to track progress

hello, most probably you are already tracking this; do you have an issue or place to track progress on the branching mechanism?

EEdu hello, most probably you are already tracking this; do you have an issue or plac...

Isaac McFadyen•11/17/23, 6:17 PM

Teams at Cloudflare don't really track progress on internal things externally, for the same reason they don't have release dates on things - if they need to push it back or rework for whatever reason they don't want to promise things a certain way or date

Isaac McFadyen•11/17/23, 6:17 PM

It's definitely being worked on or planned internally but we don't really have a way to track externally

CCyb3r-Jak3 Only if you are enterprise. The recommended way would be to run an HTTP proxy wo...

Ɇ₦ĐɆⱤV₳₥₱łⱤɆ•11/17/23, 6:19 PM

gross

dhakkad•11/17/23, 6:42 PM

I'm not sure about this but is there a way which with I can use d1 with a wordpress website?

max•11/17/23, 7:05 PM

Hi all, I'm wondering if someone can help me debug why my D1 latency is so high, or how to improve it?

Context: I'm experimenting with a very small D1 table (25 rows), and I'm doing a SELECTSELECT on an indexed unique column.

I'm seeing up to 500 - 1500ms latency doing a read from a Worker, but the console is telling me the query latency is always < 1ms. If the console is telling the truth, then the query is fast, but the network is slow.

My query latencies seem to be bimodal:

if I've made a request in the last ~minute, my observed latency is 60-120ms (lower w/ smart placement turned on)
if I haven't, my first request will be very slow, always above 500ms, often 1300ms - 1500ms

The complete code I'm measuring is:

let d1ReadResults = await env.DB.prepare("SELECT * FROM Entries WHERE hash = ?1")
    .bind(hash)
    .first();

let d1ReadResults = await env.DB.prepare("SELECT * FROM Entries WHERE hash = ?1")
    .bind(hash)
    .first();

I was expecting D1 latency from a worker to be very fast, but right now it's often slower than my network calls to services outside of Cloudflare! I'd love to improve this if I can.

Vikash•11/18/23, 1:42 PM

Is D1 sized fixed to maximum 2gb, or it can be extended on request?

If not, we are going to design our application for multiple databases instead all tables in single database.

VVikash Is D1 sized fixed to maximum 2gb, or it can be extended on request? If not, we ...

itsmatteomanf•11/18/23, 1:42 PM

It's expected to grow, but no one, at least externally, knows by how much or when.

itsmatteomanf•11/18/23, 1:43 PM

If you can split it by database, I'd split it if possible.

itsmatteomanf•11/18/23, 1:43 PM

You'd get higher performance as each one is a separate instance.

Vikash•11/18/23, 1:43 PM

I am planning my app, so I can for now.. The expected size of database is around 5gb

Vikash•11/18/23, 1:49 PM

Also any suggestion/example for bulk import? I have around 500mb CSV file needs to be imported.

busterroni•11/18/23, 6:40 PM

I think there is a bug with deleting a table that is referenced by another table. You should probably get an error that the table could not be deleted, but instead you get a timeout after a bit.

busterroni•11/18/23, 7:04 PM

Also just got this when dropping a table

{'result': [], 'success': False, 'errors': [{'code': 7500, 'message': 'Internal error in D1 storage caused object to be reset.'}], 'messages': []}

{'result': [], 'success': False, 'errors': [{'code': 7500, 'message': 'Internal error in D1 storage caused object to be reset.'}], 'messages': []}

Anon•11/19/23, 7:32 PM

good, there will be a feature or a way to achieve pagination.

AAnon good, there will be a feature or a way to achieve pagination.

Cyb3r-Jak3•11/19/23, 11:30 PM

I mean you can already do pagination. You limit your query to X amount of results and for other pages you apply an offset

user•11/20/23, 5:59 AM

https://user.you-sk.id/plCW7Cg4qgki Is there any way to get more than 2gb?

uwu

owo

SSotashi Nokomata is there any way to restore a database without overwriting? do not see any opt...

Max (@rozenmd)•11/20/23, 8:55 AM

not yet, but we plan on adding fork/clone functionality

Rick•11/20/23, 10:43 AM

hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in the future? I am not asking for any timelines, as I don't have any need right now, but I want to understand if I am locking myself into a tech choice that I may not be able to work with later. (I know that I can create multiple databases, but I fear the spreading my data over multiple databses will make things way too complicated for me)

RRick hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in th...

user•11/20/23, 2:10 PM

^^ this

Uuser https://user.you-sk.id/plCW7Cg4qgki Is there any way to get more than 2gb?

Max (@rozenmd)•11/20/23, 2:25 PM

today? no, it's a hard limit we're working on increasing

MMax (@rozenmd)today? no, it's a hard limit we're working on increasing

user•11/20/23, 3:04 PM

Is there any sort of guess on when the hard limit might be increased?

user•11/20/23, 3:04 PM

I read into it and understand that it’s a difficult task to maintain for CF

user•11/20/23, 3:04 PM

Just curious if there’s any guesses about progress

Uuser Is there any sort of guess on when the hard limit might be increased?

Max (@rozenmd)•11/20/23, 3:15 PM

we don't guess/provide estimates - once we increase the limit there will likely be a large announcement about it

RRick hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in th...

Matt Silverlock•11/20/23, 3:17 PM

Not in the next 6+ months (at least). Hyperdrive + a traditional RDBMS is likely going to be a better fit if you truly need 50GB in a single database (or have an existing DB you want to migrate) for the time being. Different use-cases.

MMax (@rozenmd)we don't guess/provide estimates - once we increase the limit there will likely ...

user•11/20/23, 3:21 PM

Alright, I assume the D1 role will be pongtsted?

clearwater4227•11/21/23, 2:07 AM

Hello all, I was looking to build with D1 but dont see a way to provision tables/schema through terraform. I just see creating databases... Am I missing something?

Cclearwater4227 Hello all, I was looking to build with D1 but dont see a way to provision table...

Cyb3r-Jak3•11/21/23, 2:33 AM

You typically wouldn't create the schema through terraform and better to use something like Drizzle (https://orm.drizzle.team/docs/quick-sqlite/d1) to manage it

Cloudflare D1 - Drizzle ORM

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

clearwater4227•11/21/23, 12:04 PM

ok, interesting....so this looks like create/define schema in js and it will create schema/tables if they dont exist?

Cclearwater4227 Hello all, I was looking to build with D1 but dont see a way to provision table...

Max (@rozenmd)•11/21/23, 12:42 PM

Generally terraform is used to spin up infrastructure, I haven't heard of folks using it for SQL commands

clearwater4227•11/21/23, 1:03 PM

I use terraform for building out my bigquery database tables. I was going to move some of my tables to D1 for performance and cost reasons so I was looking at integrating that into my terraform setup.

clearwater4227•11/21/23, 1:05 PM

kind of build tables and workers for a few tables, setup terraform setup/deploy, migrate things over, and press go. Terraform is nice for schema management as well, at least for bigquery, as I can add and change columns and the bigquery terraform integration handles the changes and migrations.

Alexey•11/21/23, 4:16 PM

hey folks. I'm new to D1 (and databases in General).
My current problem is data conversion. It seems to me I need to manually convert JS data types to SQL data types and vice versa.
For example, for D1 BOOLEAN values I get

in my function code. I expect to have

true

true

false

false

in my function code instead.
I looked into some ORMs but didn't find that functionality for automatic types conversion.

What are the ways to handle this problem? How do you approach it?

AAlexey hey folks. I'm new to D1 (and databases in General). My current problem is data ...

Bram•11/21/23, 4:30 PM

SQLite does not have a boolean storage class, but is instead stored as 0 or 1. You can use an ORM for this, I recommend

drizzle

drizzle

, a relatively new ORM that has a lot of momentum and is build with edge in mind.

Bram•11/21/23, 4:34 PM

it let's you declare sql schema's directly in typescript, where you can hint the ORM that that the integer should be interpreted as a boolean on application level: https://orm.drizzle.team/docs/column-types/sqlite integer('id', { mode: 'boolean' })integer('id', { mode: 'boolean' }), same for timestamps, json etc

SQLite column types - Drizzle ORM

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

Alexey•11/21/23, 4:35 PM

ok, interesting, will give drizzle a try. Thank for the suggestion @Bram

Unsmart•11/21/23, 5:32 PM

The main way would be to create a worker as a proxy

James•11/21/23, 5:53 PM

I think this is that same obscure issue we've seen a few times, cc @Matt: https://community.cloudflare.com/t/d1-database-is-crashed/583564

Cloudflare Community

D1 database is crashed

My D1 database is suddenly crashed. And it is not recovered. In D1 tab, there is a message “Failed to reach database. Please try again later”. What’s going on?

Cyb3r-Jak3•11/21/23, 6:49 PM

Old vs new backend. My guess is the first one hasn’t had the backend migrated.

RRed Spider Is it possible to bulk insert via the API? I have an array of items i want to in...

Hello, I’m Allie!•11/21/23, 7:53 PM

Depending on the size of the items, and whether they are rows in a single table, or multiple tables, you might be able to do a multi-insert

Hello, I’m Allie!•11/21/23, 7:54 PM

INSERT INTO table1 (column1,column2 ,..)
VALUES 
   (value1,value2 ,...),
   (value1,value2 ,...),
    ...
   (value1,value2 ,...);

INSERT INTO table1 (column1,column2 ,..)
VALUES 
   (value1,value2 ,...),
   (value1,value2 ,...),
    ...
   (value1,value2 ,...);

Hello, I’m Allie!•11/21/23, 7:54 PM

Something like this

hello, most probably you are already tracking this; do you have an issue or place to track progress

Similar Threads