are there any standardized approaches on sharding the database currently? it sucks a lot that databa

JJúlio Mühlbauer Is there some limitations on scalability compared to a postgres database? It wou...

H

Hello, I’m Allie!•2/20/24, 10:28 AM

There is https://developers.cloudflare.com/d1/platform/limits/, if that is what you mean

Bbun are there any standardized approaches on sharding the database currently? it suc...

H

Hello, I’m Allie!•2/20/24, 10:29 AM

No, though generally the optimal sharding strategy is dependent on the shape of the data you are storing

B

bunOP•2/20/24, 10:29 AM

like do i have to do 100 bindings in my wrangler.toml

Bbun like do i have to do 100 bindings in my wrangler.toml

H

Hello, I’m Allie!•2/20/24, 10:29 AM

That is a way. You can also offload large values to a separate store, if you don't need to query on them directly

HHello, I’m Allie!There is https://developers.cloudflare.com/d1/platform/limits/, if that is what ...

J

Júlio Mühlbauer•2/20/24, 10:30 AM

As I undertood, this is mostly limits impose by cloudflare that can be increased by a request or paying. But not about the technology the database uses, right?

JJúlio Mühlbauer As I undertood, this is mostly limits impose by cloudflare that can be increased...

H

Hello, I’m Allie!•2/20/24, 10:32 AM

Not necessarily. While some of the limits are applied by Cloudflare directly, others are in place because the database may not be stable above those limits. You might be able to push it farther, but CF can't guarantee that it will function correctly, so you can't raise the limit that far

HHello, I’m Allie!Not necessarily. While some of the limits are applied by Cloudflare directly, ot...

J

Júlio Mühlbauer•2/20/24, 10:32 AM

Understand. Thanks!

B

bunOP•2/20/24, 10:32 AM

are there no estimates on increasing the storage of the database?

Bbun are there no estimates on increasing the storage of the database?

H

Hello, I’m Allie!•2/20/24, 10:33 AM

Estimates of Pricing? Timeline? Other?

B

bunOP•2/20/24, 10:33 AM

timeline

Bbun timeline

H

Hello, I’m Allie!•2/20/24, 10:33 AM

:soontm:

J

Júlio Mühlbauer•2/20/24, 10:33 AM

Would love to see a postgres or something that scales made by cloudflare btw

Original message was deleted

B

Befus•2/20/24, 10:59 AM

I will have a look at this, thanks a lot!

Original message was deleted

B

Befus•2/20/24, 11:03 AM

That will be a solution to the waiting for deployment?

B

Befus•2/20/24, 11:04 AM

Awesome!

Original message was deleted

M

Max (@rozenmd)•2/20/24, 12:09 PM

here's how we load the entire northwind db (via the REST API/wrangler): https://raw.githubusercontent.com/cloudflare/d1-northwind/main/db/data-big.sql

V

Vignesh•2/20/24, 12:12 PM

Any ETA on official dynamic binding?

Original message was deleted

B

Befus•2/20/24, 12:25 PM

Gonna be trying this out these next few days first gotta setup next auth with d1

Original message was deleted

I

itsmatteomanf•2/20/24, 1:07 PM

Oh yeah, if you do a batch with reused statements especially, it’s in the 100s of thousands of inserts per call in a lot cases. Performance wise is really good.

I

itsmatteomanf•2/20/24, 1:08 PM

There are a couple of issues with reliability (but they are very rare now and they are being fixed), but for the most part it’s really good.

Original message was deleted

I

itsmatteomanf•2/20/24, 1:10 PM

That even seems slow, but it depends on what you use (Worker or HTTP API?) and the amount of data per row + number of indexes on top

Original message was deleted

I

itsmatteomanf•2/20/24, 1:36 PM

Ok, yeah, makes more sense

my rows are smaller!

Original message was deleted

H

Hello, I’m Allie!•2/20/24, 1:42 PM

Yes, but they have to be created when the script receives its first event. You can't create them before the event arrives

M

Martin Carlsson•2/20/24, 6:16 PM

Are there any way of getting data out of D1?

I mean, like aws dms, airbyte, or fivetran?

T

Thenerox•2/20/24, 11:44 PM

hello can i connect to D1 using dbeaver or something like that?

T

Tony Leung•2/21/24, 4:59 AM

Hi guys, I'm querying data with the query above, but it seems like scanning too much data. I've set index for

p.posted

p.posted

and

p.retry

p.retry

but it's still scanning the whole table

. Is there anyway to reduce scanned rows of the query above? I'm running this query every minutes so everyday I'm wasting 10-20k scanned row. This table is going to grow much larger so the waste will be more significant

Screenshot_2024-02-21_at_11.55.02_AM.png

P

prabha•2/21/24, 10:30 AM

Hi everyone.. How to create REST API for CRUD operations in D1

S

stefanoros•2/21/24, 10:45 AM

Hello everyone, i'm trying to set up a simple many-to-many join

SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;

SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;

SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;

SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;

i get this error

Error: D1_ERROR: A prepared SQL statement must contain only one statement.

Error: D1_ERROR: A prepared SQL statement must contain only one statement.

. Can't seem to figure why.

Sstefanoros Hello everyone, i'm trying to set up a simple many-to-many join ```SELECT * ...

M

Max (@rozenmd)•2/21/24, 10:56 AM

can you post the exact code, including JS/TS that you're using?

S

stefanoros•2/21/24, 10:57 AM

sure

    let result = await platform!.env.DB.prepare(
        `SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;
    `
    ).all();

    let result = await platform!.env.DB.prepare(
        `SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;
    `
    ).all();

    let result = await platform!.env.DB.prepare(
        `SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;
    `
    ).all();

    let result = await platform!.env.DB.prepare(
        `SELECT *
    FROM categories
    JOIN images_categories ON categories.id = images_categories.category_id
    JOIN images ON images_categories.image_id = images.id;
    `
    ).all();

if I remove one join statement it works

M

Max (@rozenmd)•2/21/24, 10:59 AM

that's weird - i've only seen that error with the parser failing to understand comments

M

Max (@rozenmd)•2/21/24, 11:00 AM

does it work if you combine the SQL onto a single line? like

`SELECT * FROM categories JOIN images_categories ON categories.id = images_categories.category_id JOIN images ON images_categories.image_id = images.id;`

`SELECT * FROM categories JOIN images_categories ON categories.id = images_categories.category_id JOIN images ON images_categories.image_id = images.id;`

MMax (@rozenmd)does it work if you combine the SQL onto a single line? like ``` `SELECT * FROM ...

S

stefanoros•2/21/24, 11:25 AM

Yes! It works now, thanks

TTony Leung Hi guys, I'm querying data with the query above, but it seems like scanning too ...

T

Tony Leung•2/21/24, 11:35 AM

hi guys, could anyone help me

TTony Leung Hi guys, I'm querying data with the query above, but it seems like scanning too ...

M

Max (@rozenmd)•2/21/24, 11:45 AM

is there a single index covering both

posted

posted

and

retry

retry

, or two individual indexes?

T

Tony Leung•2/21/24, 11:50 AM

yes, I'm having a composite index like this:

CREATE INDEX post_posted_retry_idx ON post (posted, retry);

CREATE INDEX post_posted_retry_idx ON post (posted, retry);

CREATE INDEX post_posted_retry_idx ON post (posted, retry);

CREATE INDEX post_posted_retry_idx ON post (posted, retry);

T

Tony Leung•2/21/24, 11:51 AM

it should cover both

posted

posted

and

retry

retry

if I place the condition in the correct order right?

T

Tony Leung•2/21/24, 11:54 AM

base on this document: "https://www.sqlite.org/optoverview.html", both

posted

posted

and

retry

retry

should be used

D

Denver James Duran•2/21/24, 12:05 PM

Hi guys! Good day to you all. Does D1 really consider 0 as NULL?

I have a NOT NULL constraint on my table.. when saving 0, it returns a

Error: D1_ERROR: FOREIGN KEY constraint failed
      at D1Database._sendOrThrow (cloudflare-internal:d1-api:66:19)

Error: D1_ERROR: FOREIGN KEY constraint failed
      at D1Database._sendOrThrow (cloudflare-internal:d1-api:66:19)

Error: D1_ERROR: FOREIGN KEY constraint failed
      at D1Database._sendOrThrow (cloudflare-internal:d1-api:66:19)

Error: D1_ERROR: FOREIGN KEY constraint failed
      at D1Database._sendOrThrow (cloudflare-internal:d1-api:66:19)

I have already logged the input, and confirmed, it is indeed 0 and typeof number.. and is absolutely NOT null... but it still results to an error..

If ever, what would be the workaround for this? I definitely would wanna save 0 but do not want a null

TThenerox hello can i connect to D1 using dbeaver or something like that?

P

prabha•2/21/24, 12:07 PM

there is a tool called DATAFLARE https://dataflare.app/ that supports D1

Dataflare

Dataflare - A simple, easy-to-use database manager

Easily connect to your Cloudflare D1, libSQL, MariaDB, MySQL, PostgreSQL, Rqlite, SQLCipher, SQLite, SQL Server databases in Dataflare, manage Table, view Data, write SQL and run Query.

Kkian SQLite supports renaming a table, adding a new column and renaming an existing c...

S

sten•2/21/24, 12:14 PM

SQLite supports a limited subset of ALTER TABLE. The ALTER TABLE command in SQLite allows these alterations of an existing table: it can be renamed; a column can be renamed; a column can be added to it; or a column can be dropped from it.

So if I want to alter the datatype of a column, I would first need to drop that column and then add it again with a new datatype?

DDenver James Duran Hi guys! Good day to you all. Does D1 really consider 0 as NULL? I have a NOT ...

M

Max (@rozenmd)•2/21/24, 12:41 PM

you're trying to insert 0 as a foreign key?

Original message was deleted

S

sten•2/21/24, 12:41 PM

There's gotta be a good way to do this though?
I guess best way is to

Rename original table with different name.
Create new table with same name as previous table have with desired column type.
Copy all previous data from old table (renamed table) to newly created table.
Drop renamed table.

MMax (@rozenmd)you're trying to insert 0 as a foreign key?

D

Denver James Duran•2/21/24, 12:43 PM

No, not a foreign key, just a regular value

CREATE TABLE IF NOT EXISTS carts (
    id INTEGER PRIMARY KEY,

    cartId INTEGER NOT NULL,
    branchId INTEGER NOT NULL,
    name TEXT NOT NULL,
    productId INTEGER NOT NULL,
    price REAL NOT NULL,  -- << HERE
    size TEXT NOT NULL,
    quantity TEXT NOT NULL,
    customerNotes TEXT NOT NULL DEFAULT '',
    discount REAL,
    discountName TEXT,
    discountDescription TEXT DEFAULT '',
    discountEndDate REAL,

    createdBy INTEGER NOT NULL,
    createdAt REAL NOT NULL,
    updatedBy INTEGER,
    updatedAt REAL,
    status INTEGER NOT NULL,
    FOREIGN KEY (branchId) REFERENCES branches(id),
    FOREIGN KEY (cartId) REFERENCES cartIds(id),
    FOREIGN KEY (productId) REFERENCES products(id),
    FOREIGN KEY (createdBy) REFERENCES users (id),
    FOREIGN KEY (updatedBy) REFERENCES users (id)
);

CREATE TABLE IF NOT EXISTS carts (
    id INTEGER PRIMARY KEY,

    cartId INTEGER NOT NULL,
    branchId INTEGER NOT NULL,
    name TEXT NOT NULL,
    productId INTEGER NOT NULL,
    price REAL NOT NULL,  -- << HERE
    size TEXT NOT NULL,
    quantity TEXT NOT NULL,
    customerNotes TEXT NOT NULL DEFAULT '',
    discount REAL,
    discountName TEXT,
    discountDescription TEXT DEFAULT '',
    discountEndDate REAL,

    createdBy INTEGER NOT NULL,
    createdAt REAL NOT NULL,
    updatedBy INTEGER,
    updatedAt REAL,
    status INTEGER NOT NULL,
    FOREIGN KEY (branchId) REFERENCES branches(id),
    FOREIGN KEY (cartId) REFERENCES cartIds(id),
    FOREIGN KEY (productId) REFERENCES products(id),
    FOREIGN KEY (createdBy) REFERENCES users (id),
    FOREIGN KEY (updatedBy) REFERENCES users (id)
);

CREATE TABLE IF NOT EXISTS carts (
    id INTEGER PRIMARY KEY,

    cartId INTEGER NOT NULL,
    branchId INTEGER NOT NULL,
    name TEXT NOT NULL,
    productId INTEGER NOT NULL,
    price REAL NOT NULL,  -- << HERE
    size TEXT NOT NULL,
    quantity TEXT NOT NULL,
    customerNotes TEXT NOT NULL DEFAULT '',
    discount REAL,
    discountName TEXT,
    discountDescription TEXT DEFAULT '',
    discountEndDate REAL,

    createdBy INTEGER NOT NULL,
    createdAt REAL NOT NULL,
    updatedBy INTEGER,
    updatedAt REAL,
    status INTEGER NOT NULL,
    FOREIGN KEY (branchId) REFERENCES branches(id),
    FOREIGN KEY (cartId) REFERENCES cartIds(id),
    FOREIGN KEY (productId) REFERENCES products(id),
    FOREIGN KEY (createdBy) REFERENCES users (id),
    FOREIGN KEY (updatedBy) REFERENCES users (id)
);

CREATE TABLE IF NOT EXISTS carts (
    id INTEGER PRIMARY KEY,

    cartId INTEGER NOT NULL,
    branchId INTEGER NOT NULL,
    name TEXT NOT NULL,
    productId INTEGER NOT NULL,
    price REAL NOT NULL,  -- << HERE
    size TEXT NOT NULL,
    quantity TEXT NOT NULL,
    customerNotes TEXT NOT NULL DEFAULT '',
    discount REAL,
    discountName TEXT,
    discountDescription TEXT DEFAULT '',
    discountEndDate REAL,

    createdBy INTEGER NOT NULL,
    createdAt REAL NOT NULL,
    updatedBy INTEGER,
    updatedAt REAL,
    status INTEGER NOT NULL,
    FOREIGN KEY (branchId) REFERENCES branches(id),
    FOREIGN KEY (cartId) REFERENCES cartIds(id),
    FOREIGN KEY (productId) REFERENCES products(id),
    FOREIGN KEY (createdBy) REFERENCES users (id),
    FOREIGN KEY (updatedBy) REFERENCES users (id)
);

DDenver James Duran No, not a foreign key, just a regular value ```sql CREATE TABLE IF NOT EXISTS ...

M

Max (@rozenmd)•2/21/24, 12:47 PM

That's not the error you're getting though:

FOREIGN KEY constraint failed

FOREIGN KEY constraint failed

D

Denver James Duran•2/21/24, 12:49 PM

that's the weird thing, the contstraint error only appears when price is 0... I wonder why..

D

Denver James Duran•2/21/24, 12:49 PM

thanks you made me realize that I may need to look at other aspects of my code

B

Befus•2/21/24, 1:15 PM

I can't seem to get drizzle to work with D1, funnily enough I believe I'm already failing at the configuration does anybody have some docs?
I have looked at the ones from Drizzle itself but I am using cloudflare pages instead of workers/wrangler

BBefus I can't seem to get drizzle to work with D1, funnily enough I believe I'm alread...

D

Denver James Duran•2/21/24, 2:02 PM

I got it to work, it's what I am using.. although I think they have limited support. I tried using it, I get trouble with JOINs.. and other queries, I end up just using the regular

D

Denver James Duran•2/21/24, 2:05 PM

Here's what I did to make it work (I am using Sveltekit deployed via cloudflare pages)

//hooks.server.ts

import { drizzle } from 'drizzle-orm/d1';

const injectD1 = async (event: RequestEvent<Partial<Record<string, string>>, string | null>) => {
    try {
        if (event.platform?.env?.DATABASE) {
            event.locals.DRIZZLE = drizzle(event.platform.env.DATABASE);
            event.locals.D1 = event.platform.env.DATABASE;
        }
        ...
    } catch (error) {
        console.log('🚀 ~ file: hooks.server.ts:27 ~ consthandle:Handle= ~ error:', error);
    }
};

//hooks.server.ts

import { drizzle } from 'drizzle-orm/d1';

const injectD1 = async (event: RequestEvent<Partial<Record<string, string>>, string | null>) => {
    try {
        if (event.platform?.env?.DATABASE) {
            event.locals.DRIZZLE = drizzle(event.platform.env.DATABASE);
            event.locals.D1 = event.platform.env.DATABASE;
        }
        ...
    } catch (error) {
        console.log('🚀 ~ file: hooks.server.ts:27 ~ consthandle:Handle= ~ error:', error);
    }
};

//hooks.server.ts

import { drizzle } from 'drizzle-orm/d1';

const injectD1 = async (event: RequestEvent<Partial<Record<string, string>>, string | null>) => {
    try {
        if (event.platform?.env?.DATABASE) {
            event.locals.DRIZZLE = drizzle(event.platform.env.DATABASE);
            event.locals.D1 = event.platform.env.DATABASE;
        }
        ...
    } catch (error) {
        console.log('🚀 ~ file: hooks.server.ts:27 ~ consthandle:Handle= ~ error:', error);
    }
};

//hooks.server.ts

import { drizzle } from 'drizzle-orm/d1';

const injectD1 = async (event: RequestEvent<Partial<Record<string, string>>, string | null>) => {
    try {
        if (event.platform?.env?.DATABASE) {
            event.locals.DRIZZLE = drizzle(event.platform.env.DATABASE);
            event.locals.D1 = event.platform.env.DATABASE;
        }
        ...
    } catch (error) {
        console.log('🚀 ~ file: hooks.server.ts:27 ~ consthandle:Handle= ~ error:', error);
    }
};

are there any standardized approaches on sharding the database currently? it sucks a lot that databa

Similar Threads

Similar Threads

Similar Threads