Hey, are you considering increasing 10gb limit in foreseen future?

Iiwgu Hey, are you considering increasing 10gb limit in foreseen future?

I

See this message for some past details from an employee - in short, maybe a bit but not really, D1 is more suited towards many small DBs (for example, one per customer) than one larger one. https://canary.discord.com/channels/595317990191398933/992060581832032316/1157239450741526560

IIsaac McFadyen See this message for some past details from an employee - in short, maybe a bit ...

I

iwguOP•8/9/24, 7:42 PM

Hey, thank you, it is clear now. In that case I guess going horizontal and requesting increase on "maximum storage per account" when limit is reached is the way.

V

Vikash•8/10/24, 1:57 PM

Yes, why not 20? For example a simple CRM may have (Accounts, Users, Apikeys, Contacts, Deals, Pipelines, Lists, SavedFilters, Tags etc) and each has account_id.

I

iska•8/10/24, 6:57 PM

hi how are d1 migrations handled in prod ? read only , no write ? for your customers, do you have a way to know time estimation it will take to let them know the duration of the maintenance ? any rollback in case something gone wrong ? what about data backup of d1 does it happen automatically ?

I

iska•8/10/24, 6:59 PM

also whats the storage limit for D1 ? curious if you have people running multiple TB at ease with it

Iiska also whats the storage limit for D1 ? curious if you have people running multipl...

I

Isaac McFadyen•8/10/24, 7:01 PM

10GB per database currently. The goal with D1 is to create many smaller DBs (i.e. one DB per customer, one DB per user, etc) rather than a single monolithic DB.

I

Isaac McFadyen•8/10/24, 7:01 PM

See here for more details: https://canary.discord.com/channels/595317990191398933/992060581832032316/1157239450741526560

I

iska•8/10/24, 7:07 PM

alright that changes thing up a bit but 1DB per customer say you have 50k users because you are a free app but only 10k are paying

you wouldnt want to create a DB for non paying users, a waste
2.having a db per customer makes aggregation queries such as grouped statistics for example a bit complicated
say you run multiples applications, you would hit rather quickly the limit of i think i saw 50k db

I

iska•8/10/24, 7:29 PM

i guess i could shard something like this:

App1:

Free Users:
- app1_free_shard_1 (1GB D1 DB)
- app1_free_shard_2 (1GB D1 DB)
  ...
Paid Users:
- app1_paid_shard_1 (5GB D1 DB)
- app1_paid_shard_2 (5GB D1 DB)
  ...
Enterprise:
- app1_tenant_bigcorp (10GB D1 DB)
- app1_tenant_shared_1 (10GB D1 DB for multiple smaller tenants)
  ...
Analytics:
- app1_analytics (10GB D1 DB)

but that includes a whole lot of infra dev and complex logic

I

iska•8/10/24, 7:30 PM

monolithic db are good because they are easy to start with , then you start sharding later in the road but starting to shard right off the boat given D1 limit seems a no brainer

wonder if someone already created some library to achieve something similar

Ccopper btw, how many secs did the entire delete in those 20 table at the end? btw2, D1 ...

V

Vikash•8/11/24, 2:13 PM

It’s resolved after creating index on all foreign keys

I

iska•8/13/24, 9:40 AM

surely, i wish there was a library that offered the orchestration of sharding D1 dbs, with features like auto resharding, migration sharding, visual interface , would be so valuable to workers D1 eco system

H

hendrixString•8/13/24, 11:49 AM

Hi there, a quick question. Why doesn't the REST-API of D1 enables a batch of parameterized statements (it only allows static statements joined by a semicolon, which is SQL Injection vulnerable). Any plans as to when this will be supported ? The way it is supported with Workers

S

sandulat•8/13/24, 1:31 PM

Hey! I'm running a simple SQL query with D1 (paid plan) and Drizzle every minute in a Worker's CRON job:

await db
    .delete(schema.authRequests)
    .where(
      lte(
        schema.authRequests.createdAt,
        dateToUnixepoch(subMinutes(new Date(), 3)),
      ),
    );

await db
    .delete(schema.authRequests)
    .where(
      lte(
        schema.authRequests.createdAt,
        dateToUnixepoch(subMinutes(new Date(), 3)),
      ),
    );

await db
    .delete(schema.authRequests)
    .where(
      lte(
        schema.authRequests.createdAt,
        dateToUnixepoch(subMinutes(new Date(), 3)),
      ),
    );

await db
    .delete(schema.authRequests)
    .where(
      lte(
        schema.authRequests.createdAt,
        dateToUnixepoch(subMinutes(new Date(), 3)),
      ),
    );

The query has been running thousands of times without any issues, but today I encountered this random error:

D1_ERROR: D1 storage operation exceeded timeout which caused the object to be reset.

D1_ERROR: D1 storage operation exceeded timeout which caused the object to be reset.

The

authRequests

authRequests

table is empty 99% of the time, or occasionally contains just a few rows (<20). Could this be an issue on Cloudflare's side?

Here's the

createdAt

createdAt

column definition:

  createdAt: integer("created_at")
    .default(sql`(unixepoch())`)
    .notNull()

  createdAt: integer("created_at")
    .default(sql`(unixepoch())`)
    .notNull()

  createdAt: integer("created_at")
    .default(sql`(unixepoch())`)
    .notNull()

  createdAt: integer("created_at")
    .default(sql`(unixepoch())`)
    .notNull()

It has no indexes, but I find it hard to believe that's the problem.

A

Abensur•8/13/24, 9:51 PM

Anyone tried the request form to increase "Maximum characters (bytes) in a LIKE or GLOB pattern"?

A

Abensur•8/13/24, 9:52 PM

Maybe I should use eq instead of like

A

Abensur•8/13/24, 9:53 PM

yeah, nvm, eq worked just fine, wrong use case for like

0

0xwei_hup•8/14/24, 5:25 AM

--preview boolean optional
Execute commands/files against a preview D1 database (as defined by preview_database_id in Wrangler.toml).

--local boolean(default: true) optional
Execute commands/files against a local database for use with wrangler dev.

--preview boolean optional
Execute commands/files against a preview D1 database (as defined by preview_database_id in Wrangler.toml).

--local boolean(default: true) optional
Execute commands/files against a local database for use with wrangler dev.

--preview boolean optional
Execute commands/files against a preview D1 database (as defined by preview_database_id in Wrangler.toml).

--local boolean(default: true) optional
Execute commands/files against a local database for use with wrangler dev.

--preview boolean optional
Execute commands/files against a preview D1 database (as defined by preview_database_id in Wrangler.toml).

--local boolean(default: true) optional
Execute commands/files against a local database for use with wrangler dev.

hey guys, would like to ask what's the difference for local and preview actually? I only know that local is stored at your local

.wrangler/state/v3/d1/miniflare-d1databaseobject/...

.wrangler/state/v3/d1/miniflare-d1databaseobject/...

directory but what's preview?

I

iska•8/14/24, 6:33 AM

Is D1 going to support auto scaling some day so user doesn’t have to deal manually with sharding ? Like how long can we estimate this feature to be ready approximately less than a year from now or no eta at all ?

00xwei_hup ```--preview boolean optional Execute commands/files against a preview D1 databa...

H

Hello, I’m Allie!•8/14/24, 6:01 PM

Preview runs against a real D1 DB

J

jacob•8/14/24, 6:04 PM

Are D1 errors just presented as a regular "Error"? How should I handle a scenario where I want to check specifically if the error in my catch block is due to a unique constraint failing

J

jacob•8/14/24, 6:06 PM

I'm effectively trying to do something like this ..

try {
  await this.LandmarkRepository.addLandmark(label, locations[0]);
} catch (e) {
  if (e instanceof SomeUniqueConstraintError) {
            
  }
}

try {
  await this.LandmarkRepository.addLandmark(label, locations[0]);
} catch (e) {
  if (e instanceof SomeUniqueConstraintError) {
            
  }
}

try {
  await this.LandmarkRepository.addLandmark(label, locations[0]);
} catch (e) {
  if (e instanceof SomeUniqueConstraintError) {
            
  }
}

try {
  await this.LandmarkRepository.addLandmark(label, locations[0]);
} catch (e) {
  if (e instanceof SomeUniqueConstraintError) {
            
  }
}

A

arch•8/14/24, 8:32 PM

How do people prefer to manage their Workers + D1 schemas? So far I've been relying on just SQL files I manually load in. Prefer not using an ORM in most cases :p

J

jdruwe•8/14/24, 9:58 PM

Guys, I have a question regarding pushing a lot of data into D1. Basically I want to insert around 300K records (based on data from a CSV file I am passed). I build a UI using Hono inside a worker but I am running into the following error: "Exceeded CPU Time Limits", so to me it seems like a worker is not the best solution. What other option can you suggest to me? Could I just insert the data using the REST api? I just noticed there is also an option for: https://developers.cloudflare.com/workers/testing/local-development/#develop-using-remote-resources-and-bindings. It also results the same error, isn't that weird as I am executing the logic locally? For now I think something like: $ npx wrangler d1 execute example-db --remote --file=users_export.sql is better

A

Ashish Rawat•8/15/24, 7:15 AM

D1 is not usable without sharding. Just for mini projects it's good. I want to know is there any lib to shard d1 with workers.

AAshish Rawat D1 is not usable without sharding. Just for mini projects it's good. I want to k...

I

iska•8/15/24, 9:24 AM

im using drizzle orm to avoid vendor lock in and will move to aws rds whenever i hit 10 gb limit, dealing with sharding is a hassle , will wait for an auto scale solution from cloudflare

I

iska•8/15/24, 9:24 AM

i m building a product , dont want to deal with infra too much

R

rossanmol•8/15/24, 5:26 PM

Getting this error not authorized: SQLITE_AUTH in Workers when using D1? Anyone knows what it could be?

Rrossanmol Getting this error not authorized: SQLITE_AUTH in Workers when using D1? Anyone ...

I

Isaac McFadyen•8/15/24, 5:27 PM

Are you trying to access internal SQLite tables or use SQLite pragmas?

I

Isaac McFadyen•8/15/24, 5:27 PM

Some are blocked on D1 for security reasons.

I

Isaac McFadyen•8/15/24, 5:28 PM

If you are using a database ORM or explorer then it's possible it's making those queries on your behalf to try and inspect the schema of the database (and getting blocked).

J

jdruwe•8/15/24, 9:49 PM

When importing data into a D1 db using the wrangler command it mentions: "This process may take some time, during which your D1 database will be unavailable to serve queries.". I noticed this also in a test as drizzle showed "7500: Currently processing a long-running import." on a select query. Is there any way to get around this? Lets assume that the DB is already being used in production, I would like for the app to continue working. Any suggestions?

O

oof2win2•8/16/24, 5:38 AM

is there any eta on read replicas with d1?

Y

YuryC•8/16/24, 6:48 PM

Hello all! I need to rename a few columns in the existing table. For some reason, the RENAME statement doesn't work for me. Does d1 support the RENAME statement? Thanks in advance.

YYuryC Hello all! I need to rename a few columns in the existing table. For some reason...

I

Isaac McFadyen•8/16/24, 6:50 PM

D1 is SQLite, so see https://stackoverflow.com/questions/805363/how-do-i-rename-a-column-in-an-sqlite-database-table

Stack Overflow

How do I rename a column in an SQLite database table?

I need to rename table columns in an SQLite database. This similar question doesn't mention SQLite. From the documentation of ALTER TABLE I gather that it's not a single statement.
What is a generi...

S

Steven•8/16/24, 7:07 PM

Hi all,
I have an issue I don't understand: when i save values to a boolean column they seem to be saved as strings (true, false, or in the case of accidental error in inserting some random other strings).

This is on a local d1 database in development.

I was under the impression that values would be converted to 0 or 1 automatically (or error thrown if not truthy or falsy). Is this not correct?

SSteven Hi all, I have an issue I don't understand: when i save values to a boolean colu...

I

Isaac McFadyen•8/16/24, 7:12 PM

So yes,

true

true

and

false

false

will be cooerced into integers, but SQLite (which D1 uses) also has the concept of "flexible datatypes" that other DB engines don't have: https://sqlite.org/datatype3.html

I

Isaac McFadyen•8/16/24, 7:12 PM

It means you can insert a string into an integer column, or an integer into a string column, and the DB engine will just accept it

I

Isaac McFadyen•8/16/24, 7:13 PM

You can use STRICT tables if you want datatypes enforced: https://sqlite.org/stricttables.html

S

Steven•8/16/24, 7:14 PM

mm, okay. Thanks. Good to know.

But I do have 'true' and 'false' values in my boolean column. Those should be converted to 1 & 0 then right? Even without STRICT

I

Isaac McFadyen•8/16/24, 7:15 PM

I'm not sure if it's case-sensitive or not?

S

Steven•8/16/24, 7:18 PM

mm, i'll do a typecheck. maybe it's getting turned into a string somewhere

Hey, are you considering increasing 10gb limit in foreseen future?

Similar Threads

Similar Threads

Similar Threads