is the 500MB table size a hard limit atm ? - Cloudflare Developers

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

Ian

I

IanOP•8/27/23, 12:58 AM

Looking for at least 10GB Copium

Copium

Ian

IIan Looking for at least 10GB <:Copium:839843826650972182>

chientrm.com

C

chientrm.com•8/27/23, 1:07 AM

maybe snowflake id can help you by spliting data into multiple database (500 MB each)

Ian

IIan is the 500MB table size a hard limit atm ?

Cyb3r-Jak3

C

Cyb3r-Jak3•8/27/23, 1:08 AM

It is a DB size not table size. Right now it is a hard limit and only reported increase size is to 1 GB per DB

chientrm.com

Cchientrm.com maybe snowflake id can help you by spliting data into multiple database (500 MB ...

Ian

I

IanOP•8/27/23, 1:09 AM

It's .. gonna be a pain, I have to bind every db

Cyb3r-Jak3

CCyb3r-Jak3 It is a DB size not table size. Right now it is a hard limit and only reported i...

Ian

I

IanOP•8/27/23, 1:09 AM

yikes ...

Ian

IIan It's .. gonna be a pain, I have to bind every db

chientrm.com

C

chientrm.com•8/27/23, 1:09 AM

you can bind databases into an array

Ian

I

IanOP•8/27/23, 1:10 AM

max is 10

Ian

chientrm.com

C

chientrm.com•8/27/23, 1:10 AM

is the 10GB required by now or in the future?

Ian

I

IanOP•8/27/23, 1:11 AM

now I need at least 3gb, and it is expected to grow to 100gb+ in the future

Ian

I

IanOP•8/27/23, 1:11 AM

sweatguy

Ian

IIan now I need at least 3gb, and it is expected to grow to 100gb+ in the future

chientrm.com

C

chientrm.com•8/27/23, 1:11 AM

that must be a huge database for a huge project.

Ian

I

IanOP•8/27/23, 1:12 AM

Yeh, a lot of data

Ian

I

IanOP•8/27/23, 1:13 AM

Well I guess I can't put that usecase on d1 for now

Ian

IIan Well I guess I can't put that usecase on d1 for now

chientrm.com

C

chientrm.com•8/27/23, 1:33 AM

didn't see that limit of 10 databases

Ian

I

IanOP•8/27/23, 1:34 AM

either way i am not that happy about splitting data into multiple dbs tbh

Ian

IIan either way i am not that happy about splitting data into multiple dbs tbh

chientrm.com

C

chientrm.com•8/27/23, 1:35 AM

in case you run out of D1 database, you can use this package and self-host more d1: https://github.com/chientrm/wrangler-proxy

chientrm.com

C

chientrm.com•8/27/23, 1:35 AM

by running wrangler at local mode.

Ian

I

IanOP•8/27/23, 1:36 AM

I would spin postgresql if I want the db to be hosted on something Copium

Copium

Ian

IIan I would spin postgresql if I want the db to be hosted on something <:Copium:8398...

chientrm.com

C

chientrm.com•8/27/23, 1:38 AM

yeh me same.

chientrm.com

C

chientrm.com•8/27/23, 1:43 AM

I need suggestion on snowflake id alternative for serverless runtime

Cyb3r-Jak3

C

Cyb3r-Jak3•8/27/23, 1:44 AM

Yeah something with 100 GB needed isn't going to be easy to host

Matt Silverlock

MMatt Silverlock There is no support for downloading backups at this point. You don’t need to wit...

mukhtharcm

M

mukhtharcm•8/27/23, 12:34 PM

Any hope for this download option coming soon?

mukhtharcm

Mmukhtharcm Any hope for this download option coming soon?

Matt Silverlock

M

Matt Silverlock•8/27/23, 12:46 PM

It won’t be soon. What is your goal? To backup outside? To run locally? Something else?

chientrm.com

C

chientrm.com•8/27/23, 1:53 PM

Wish for the increment of 10 D1 databases limit

JThiel

J

JThiel•8/27/23, 3:05 PM

Hey there, kind of cross-posting this as I posted this already in #next-on-pages. Yet I am unsure if this is actually a D1 or Pages / Next issue. We encouter the following error on a deployed app only. No issues locally with Wrangler (3.5.0) and the same app code:

"Error: D1_TYPE_ERROR: Type 'object' not supported for value 'Tue Sep 26 2023 12:48:07 GMT+0000 (Coordinated Universal Time)'
    at D1PreparedStatement.bind (bundledWorker-0.17018918700489616.mjs:898:17)
    at s.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:12:44988)
    at __next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:48052
    at Ii.provideConnection (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:83082)
    at async at.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:47999)
    at async _e.execute (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19677)
    at async _e.executeTakeFirst (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19900)
    at async _e.executeTakeFirstOrThrow (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19972)
    at async createSession (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:91101)
    at async Ue.<computed> (__next-on-pages-dist__/webpack/71992532242772f20ff0087800cdd46e.js:56:78937)"

"Error: D1_TYPE_ERROR: Type 'object' not supported for value 'Tue Sep 26 2023 12:48:07 GMT+0000 (Coordinated Universal Time)'
    at D1PreparedStatement.bind (bundledWorker-0.17018918700489616.mjs:898:17)
    at s.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:12:44988)
    at __next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:48052
    at Ii.provideConnection (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:83082)
    at async at.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:47999)
    at async _e.execute (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19677)
    at async _e.executeTakeFirst (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19900)
    at async _e.executeTakeFirstOrThrow (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19972)
    at async createSession (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:91101)
    at async Ue.<computed> (__next-on-pages-dist__/webpack/71992532242772f20ff0087800cdd46e.js:56:78937)"

SQL Schema

CREATE TABLE "Session"
(
    "id"           INTEGER  NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
    "sessionToken" TEXT     NOT NULL,
    "userId"       TEXT     NOT NULL,
    "expires"      DATETIME NOT NULL,
    FOREIGN KEY ("userId") REFERENCES "User" ("id") ON DELETE RESTRICT ON UPDATE CASCADE
);

CREATE TABLE "Session"
(
    "id"           INTEGER  NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
    "sessionToken" TEXT     NOT NULL,
    "userId"       TEXT     NOT NULL,
    "expires"      DATETIME NOT NULL,
    FOREIGN KEY ("userId") REFERENCES "User" ("id") ON DELETE RESTRICT ON UPDATE CASCADE
);

DB Type for Kysely (with D1Dialect)

export interface Database {
...
    Session: {
        id: GeneratedAlways<string>
        userId: string
        sessionToken: string
        expires: string
    };
...
}

export interface Database {
...
    Session: {
        id: GeneratedAlways<string>
        userId: string
        sessionToken: string
        expires: string
    };
...
}

Any obvious fails on our end?

Matt Silverlock

MMatt Silverlock It won’t be soon. What is your goal? To backup outside? To run locally? Somethin...

mukhtharcm

M

mukhtharcm•8/27/23, 3:11 PM

It was to run locally.

I have kind of complex setup for my dev.

I'm using kysely for using d1,

when developing locally, I connect it to a local sqlite file using better-sqlite

mukhtharcm

M

mukhtharcm•8/27/23, 3:11 PM

I was thinking, if we could download the db, I can get the production data on local

mukhtharcm

M

mukhtharcm•8/27/23, 3:12 PM

I think it was available before. I remember downloading it in an old prject

dominik-mukrecki

D

dominik-mukrecki•8/28/23, 11:05 AM

Hi, any plans to add postgres compatibility mode into D1? Regards

dominik-mukrecki

Ddominik-mukrecki Hi, any plans to add postgres compatibility mode into D1? Regards 🙂

Matt Silverlock

M

Matt Silverlock•8/28/23, 11:53 AM

No plans. None of the “Postgres compatibility” modes work very well — there are too many differences. ORMs don’t work well. You will find yourself frustrated debugging the differences. You can’t just copy queries over.

dominik-mukrecki

Ddominik-mukrecki Hi, any plans to add postgres compatibility mode into D1? Regards 🙂

Matt Silverlock

M

Matt Silverlock•8/28/23, 11:53 AM

Instead: what about Postgres do you actually need?

Matt Silverlock

MMatt Silverlock Instead: what about Postgres do you actually need?

dominik-mukrecki

D

dominik-mukrecki•8/28/23, 12:00 PM

I asked because of Postgres management is sometimes frustrated and there is no ideal managed or self-hosted solution. Drop-in replacement will be great for many applications, something like Supabase, where the legacy application could be connected directly to the DB and changes are monitored in the realtime, with the ability to track changes for example.

dominik-mukrecki

Ddominik-mukrecki I asked because of Postgres management is sometimes frustrated and there is no i...

Matt Silverlock

M

Matt Silverlock•8/28/23, 12:40 PM

Broadcasting real time changes doesn’t require Postgres though — D1 could certainly do that at some point

EvanPiro

E

EvanPiro•8/28/23, 5:36 PM

How unreliable is D1 currently? I'm currently developing an application that will be live in the next few weeks and am strongly considering using D1 for the backend and dealing with alpha issues as they come.

EvanPiro

EEvanPiro How unreliable is D1 currently? I'm currently developing an application that wil...

Matt Silverlock

M

Matt Silverlock•8/28/23, 5:44 PM

It’s alpha, but… we’re pretty close to beta. Most folks don’t encounter any hard errors: we’re addressing the remaining transient ones we see. Lots of folks running it day to day. But I have the responsibility of saying “beta”.

EvanPiro

E

EvanPiro•8/28/23, 6:00 PM

That's great to hear! I'm trying to move all my go-to solutions from AWS to Cloudflare. The next one on my list is Cognito. Do you have a guide or some docs on such a migration / feature parity with AWS?

EvanPiro

EEvanPiro That's great to hear! I'm trying to move all my go-to solutions from AWS to Clou...

Matt Silverlock

M

Matt Silverlock•8/28/23, 6:02 PM

What are you using Cognito for? For internal users: Cloudflare Access (see the public docs) is the go-to. We don’t have an “external” user-facing auth product but Clerk (clerk.dev) is fantastic.

Matt Silverlock

MMatt Silverlock What are you using Cognito for? For internal users: Cloudflare Access (see the p...

EvanPiro

E

EvanPiro•8/28/23, 6:06 PM

I'm using it for exactly what Clerk does : ). Thanks for the pointer!

EvanPiro

E

EvanPiro•8/28/23, 6:11 PM

Is there a solution similar to AWS CloudFormation where I can export / import / diff all of the config of my cloudflare account? Ideally I'd like to have all the settings in my account as versioned YML and rely on CI to sync changes.

EvanPiro

E

EvanPiro•8/28/23, 6:12 PM

Apologies if this is getting off topic

Unsmart

U

Unsmart•8/28/23, 6:15 PM

Cloudflare doesnt really have any custom thing for that. Best bet is either terraform or pulumi... pulumi is probably easier to use but not sure how much it supports

Matt Silverlock

MMatt Silverlock It’s alpha, but… we’re pretty close to beta. Most folks don’t encounter any hard...

Rabid

R

Rabid•8/28/23, 7:33 PM

What kind of reasonable limits do you forsee? I know the limit is going to increase from 500 MB in beta, but do you see this increasing to something like 1 GB? 10 GB? 100 GB? I don't need it to be enormous, but just want to make sure I'm not going to kneecap myself in the future if my app gets a decent amount of users.

Rabid

RRabid What kind of reasonable limits do you forsee? I know the limit is going to incr...

chientrm.com

C

chientrm.com•8/28/23, 7:34 PM

you can self-hosted D1

chientrm.com

Cchientrm.com you can self-hosted D1

Hello, I’m Allie!

H

Hello, I’m Allie!•8/28/23, 9:55 PM

I mean, if you are going to self-host anyway, then you might as well use a DB built for larger sizes

Hello, I’m Allie!

H

Hello, I’m Allie!•8/28/23, 9:56 PM

You might be able to push 100 GB through local D1, but why risk it when you can just spin up Postgres or MySQL

Rabid

R

Rabid•8/28/23, 10:21 PM

Yeah if I was going to self host I would just use postgres, or just stick with RDS

"Error: D1_TYPE_ERROR: Type 'object' not supported for value 'Tue Sep 26 2023 12:48:07 GMT+0000 (Coordinated Universal Time)'
    at D1PreparedStatement.bind (bundledWorker-0.17018918700489616.mjs:898:17)
    at s.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:12:44988)
    at __next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:48052
    at Ii.provideConnection (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:83082)
    at async at.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:47999)
    at async _e.execute (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19677)
    at async _e.executeTakeFirst (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19900)
    at async _e.executeTakeFirstOrThrow (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19972)
    at async createSession (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:91101)
    at async Ue.<computed> (__next-on-pages-dist__/webpack/71992532242772f20ff0087800cdd46e.js:56:78937)"

"Error: D1_TYPE_ERROR: Type 'object' not supported for value 'Tue Sep 26 2023 12:48:07 GMT+0000 (Coordinated Universal Time)'
    at D1PreparedStatement.bind (bundledWorker-0.17018918700489616.mjs:898:17)
    at s.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:12:44988)
    at __next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:48052
    at Ii.provideConnection (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:83082)
    at async at.executeQuery (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:47999)
    at async _e.execute (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19677)
    at async _e.executeTakeFirst (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19900)
    at async _e.executeTakeFirstOrThrow (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:19972)
    at async createSession (__next-on-pages-dist__/webpack/9b3bdf9eee839442fe37ed586efaff1d.js:1:91101)
    at async Ue.<computed> (__next-on-pages-dist__/webpack/71992532242772f20ff0087800cdd46e.js:56:78937)"

CREATE TABLE "Session"
(
    "id"           INTEGER  NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
    "sessionToken" TEXT     NOT NULL,
    "userId"       TEXT     NOT NULL,
    "expires"      DATETIME NOT NULL,
    FOREIGN KEY ("userId") REFERENCES "User" ("id") ON DELETE RESTRICT ON UPDATE CASCADE
);

CREATE TABLE "Session"
(
    "id"           INTEGER  NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
    "sessionToken" TEXT     NOT NULL,
    "userId"       TEXT     NOT NULL,
    "expires"      DATETIME NOT NULL,
    FOREIGN KEY ("userId") REFERENCES "User" ("id") ON DELETE RESTRICT ON UPDATE CASCADE
);