The data will change often, prices and availability change

Bbusterroni The data will change often, prices and availability change

_

_tweak^prone•11/16/23, 4:04 PM

I might suspect that one DB per airport would result in rather a bit of a mess when doing lookups cross a few airports (?)

Bbusterroni Hi, I'm working on a flight search engine and am considering D1. I'm concerned t...

H

Hello, I’m Allie!•11/16/23, 4:46 PM

As a mockup, try pulling that data into an SQLite file, and see how big it is

H

Hello, I’m Allie!•11/16/23, 4:46 PM

You might be able to use certain techniques to minify it a tad

B

busterroniOP•11/16/23, 4:56 PM

It's going to be well over 2GB, there are millions of flights a year and a lot of updates

Bbusterroni It's going to be well over 2GB, there are millions of flights a year and a lot o...

S

Sam•11/16/23, 7:09 PM

Would recommend you using a Postgres cluster with CF Hyperdrive

Ttadeumaia Hello. Any news on this or maybe some alpha/beta/waitlist? This is holding me to...

J

João Castro•11/16/23, 9:08 PM

+1, it would be a huge improvement to have D1 in South America

U

Unsmart•11/17/23, 1:31 AM

D1 has never really had many downtime issues tbh beyond maybe weird edge cases for a very very few amount of people.

C

Cyb3r-Jak3•11/17/23, 2:52 AM

The API is always going to be slower. I would recommend making a proxy worker

C

Cyb3r-Jak3•11/17/23, 3:22 AM

The API is always going to be slower due to how the auth is handled

SSotashi Nokomata would be nice to have D1's HTTP API reduce the latency, for simple use cases dir...

M

Max (@rozenmd)•11/17/23, 9:16 AM

actually working on that

V

VIO•11/17/23, 12:21 PM

I just hit the 2GB limit

V

VIO•11/17/23, 12:22 PM

is there anyway to get a dumb of my db

V

VIO•11/17/23, 12:22 PM

?

M

Max (@rozenmd)•11/17/23, 12:55 PM

https://github.com/nora-soderlund/cloudflare-d1-backups until we build it natively

M

maguayo•11/17/23, 1:44 PM

Is D1 down?
Error: D1_TYPE_ERROR: Type 'undefined' not supported for value 'undefined'

K

kian•11/17/23, 1:47 PM

That isn't an issue with D1,

undefined

undefined

has never been supported.

K

kian•11/17/23, 1:47 PM

https://developers.cloudflare.com/d1/platform/client-api/#type-conversion

Not supported. Queries with undefined values will return a D1_TYPE_ERROR

K

kian•11/17/23, 1:47 PM

Make sure you're not passing

undefined

undefined

to

bind

bind

Kkian That isn't an issue with D1, `undefined` has never been supported.

M

maguayo•11/17/23, 1:48 PM

The app was made 1 year ago, did not change anything, but it's failing today

K

kian•11/17/23, 1:50 PM

I can only recommend logging out the parameters you are passing to

bind

bind

and seeing which one is undefined, and going from there - the part of the D1 binding that throws that error hasn't changed functionally in a very long time.

J

jiangok•11/17/23, 5:17 PM

hi, my CF page production cannot find the d1 database while preview can:

[env.staging]
d1_databases = [
    { binding = "DB", database_name = "test1", database_id = "1111", preview_database_id = "DB" },
]

wrangler pages deploy ./public --project-name myproj --env staging  # works

[env.production]
d1_databases = [
    { binding = "DB", database_name = "test1-prod", database_id = "1111" },
]

wrangler pages deploy ./public --project-name myproj --env production --branch main # throw "Cannot read properties of undefined (reading 'prepare')"

[env.staging]
d1_databases = [
    { binding = "DB", database_name = "test1", database_id = "1111", preview_database_id = "DB" },
]

wrangler pages deploy ./public --project-name myproj --env staging  # works

[env.production]
d1_databases = [
    { binding = "DB", database_name = "test1-prod", database_id = "1111" },
]

wrangler pages deploy ./public --project-name myproj --env production --branch main # throw "Cannot read properties of undefined (reading 'prepare')"

[env.staging]
d1_databases = [
    { binding = "DB", database_name = "test1", database_id = "1111", preview_database_id = "DB" },
]

wrangler pages deploy ./public --project-name myproj --env staging  # works

[env.production]
d1_databases = [
    { binding = "DB", database_name = "test1-prod", database_id = "1111" },
]

wrangler pages deploy ./public --project-name myproj --env production --branch main # throw "Cannot read properties of undefined (reading 'prepare')"

[env.staging]
d1_databases = [
    { binding = "DB", database_name = "test1", database_id = "1111", preview_database_id = "DB" },
]

wrangler pages deploy ./public --project-name myproj --env staging  # works

[env.production]
d1_databases = [
    { binding = "DB", database_name = "test1-prod", database_id = "1111" },
]

wrangler pages deploy ./public --project-name myproj --env production --branch main # throw "Cannot read properties of undefined (reading 'prepare')"

Any idea? Thanks very much!

M

Matt Silverlock•11/17/23, 5:22 PM

Pages doesn’t use

wrangler.toml

wrangler.toml

in production: https://developers.cloudflare.com/pages/platform/functions/bindings/#d1-databases

Bindings · Cloudflare Pages docs

A binding enables your Pages Functions to interact with resources on the Cloudflare developer platform. Use bindings to integrate your Pages Functions …

E

Edu•11/17/23, 6:16 PM

hello, most probably you are already tracking this; do you have an issue or place to track progress on the branching mechanism?

EEdu hello, most probably you are already tracking this; do you have an issue or plac...

I

Isaac McFadyen•11/17/23, 6:17 PM

Teams at Cloudflare don't really track progress on internal things externally, for the same reason they don't have release dates on things - if they need to push it back or rework for whatever reason they don't want to promise things a certain way or date

I

Isaac McFadyen•11/17/23, 6:17 PM

It's definitely being worked on or planned internally but we don't really have a way to track externally

CCyb3r-Jak3 Only if you are enterprise. The recommended way would be to run an HTTP proxy wo...

Ɇ

Ɇ₦ĐɆⱤV₳₥₱łⱤɆ•11/17/23, 6:19 PM

gross

D

dhakkad•11/17/23, 6:42 PM

I'm not sure about this but is there a way which with I can use d1 with a wordpress website?

M

max•11/17/23, 7:05 PM

Hi all, I'm wondering if someone can help me debug why my D1 latency is so high, or how to improve it?

Context: I'm experimenting with a very small D1 table (25 rows), and I'm doing a

SELECT

SELECT

on an indexed unique column.

I'm seeing up to 500 - 1500ms latency doing a read from a Worker, but the console is telling me the query latency is always < 1ms. If the console is telling the truth, then the query is fast, but the network is slow.

My query latencies seem to be bimodal:

if I've made a request in the last ~minute, my observed latency is 60-120ms (lower w/ smart placement turned on)
if I haven't, my first request will be very slow, always above 500ms, often 1300ms - 1500ms

The complete code I'm measuring is:

let d1ReadResults = await env.DB.prepare("SELECT * FROM Entries WHERE hash = ?1")
    .bind(hash)
    .first();

let d1ReadResults = await env.DB.prepare("SELECT * FROM Entries WHERE hash = ?1")
    .bind(hash)
    .first();

let d1ReadResults = await env.DB.prepare("SELECT * FROM Entries WHERE hash = ?1")
    .bind(hash)
    .first();

let d1ReadResults = await env.DB.prepare("SELECT * FROM Entries WHERE hash = ?1")
    .bind(hash)
    .first();

I was expecting D1 latency from a worker to be very fast, but right now it's often slower than my network calls to services outside of Cloudflare! I'd love to improve this if I can.

V

Vikash•11/18/23, 1:42 PM

Is D1 sized fixed to maximum 2gb, or it can be extended on request?

If not, we are going to design our application for multiple databases instead all tables in single database.

VVikash Is D1 sized fixed to maximum 2gb, or it can be extended on request? If not, we ...

I

itsmatteomanf•11/18/23, 1:42 PM

It's expected to grow, but no one, at least externally, knows by how much or when.

I

itsmatteomanf•11/18/23, 1:43 PM

If you can split it by database, I'd split it if possible.

I

itsmatteomanf•11/18/23, 1:43 PM

You'd get higher performance as each one is a separate instance.

V

Vikash•11/18/23, 1:43 PM

I am planning my app, so I can for now.. The expected size of database is around 5gb

V

Vikash•11/18/23, 1:49 PM

Also any suggestion/example for bulk import? I have around 500mb CSV file needs to be imported.

B

busterroniOP•11/18/23, 6:40 PM

I think there is a bug with deleting a table that is referenced by another table. You should probably get an error that the table could not be deleted, but instead you get a timeout after a bit.

B

busterroniOP•11/18/23, 7:04 PM

Also just got this when dropping a table

{'result': [], 'success': False, 'errors': [{'code': 7500, 'message': 'Internal error in D1 storage caused object to be reset.'}], 'messages': []}

{'result': [], 'success': False, 'errors': [{'code': 7500, 'message': 'Internal error in D1 storage caused object to be reset.'}], 'messages': []}

{'result': [], 'success': False, 'errors': [{'code': 7500, 'message': 'Internal error in D1 storage caused object to be reset.'}], 'messages': []}

{'result': [], 'success': False, 'errors': [{'code': 7500, 'message': 'Internal error in D1 storage caused object to be reset.'}], 'messages': []}

A

Anon•11/19/23, 7:32 PM

good, there will be a feature or a way to achieve pagination.

AAnon good, there will be a feature or a way to achieve pagination.

C

Cyb3r-Jak3•11/19/23, 11:30 PM

I mean you can already do pagination. You limit your query to X amount of results and for other pages you apply an offset

U

user•11/20/23, 5:59 AM

https://user.you-sk.id/plCW7Cg4qgki Is there any way to get more than 2gb?

uwu

owo

SSotashi Nokomata is there any way to restore a database without overwriting? do not see any opt...

M

Max (@rozenmd)•11/20/23, 8:55 AM

not yet, but we plan on adding fork/clone functionality

R

Rick•11/20/23, 10:43 AM

hey CF folks. Do we expect 50Gb + storage size for individual d1 databases in the future? I am not asking for any timelines, as I don't have any need right now, but I want to understand if I am locking myself into a tech choice that I may not be able to work with later. (I know that I can create multiple databases, but I fear the spreading my data over multiple databses will make things way too complicated for me)

The data will change often, prices and availability change

Similar Threads

Similar Threads

Similar Threads