Do you known if this soft limit of 50,000 databases will be increased again later for the paid plans

Do you known if this soft limit of 50,000 databases will be increased again later for the paid plans please?
I known it's possible to have kind of unlimited databases in WfP, but this require a business account of $200 / month if I understood correctly.
Our goal is to isolate 1 DB per user for security reason (like a sandbox)

GGuillaumeLakano Do you known if this soft limit of 50,000 databases will be increased again late...

itsmatteomanf•10/3/23, 9:36 AM

Matt will reply specifically for this case, but in general for Cloudflare soft limits can be increased for any account, but may require specific subscriptions to do so. I presume, but have no confirmation, that you can increase it just by asking Matt (while in beta, maybe even after) or opening a support ticket.

GGuillaumeLakano Do you known if this soft limit of 50,000 databases will be increased again late...

Epailes•10/3/23, 9:47 AM

Fyi ent contracts are closer to $2000/month last I checked

EEpailes Fyi ent contracts are closer to $2000/month last I checked

GuillaumeLakanoOP•10/3/23, 9:48 AM

There is a Business plan at $200, but I don't known if WfP is only for Enterprises account or Business, but in all case even Business for us it's out of our budget

GuillaumeLakanoOP•10/3/23, 9:50 AM

Generally, the security features of Cloudflare are free, as the goal is to have a better/safer internet. Our goal is only to add a extra security layer with isolate each DB per user, so in case of a successful attack, it's mitigated to only the user database

divby0•10/3/23, 10:00 AM

I upgraded my wrangler v2 to v3 and in that process I start getting

Expression tree is too large (maximum depth 20)

Expression tree is too large (maximum depth 20)

for some queries. Those are not particularly complex, it's just like get x -> join this field -> join this field, so I'd guess this would be the depth 2 or 3?

It worked in v2 and it works on prod, so how comes it doesn't work when I locally upgrade to wrangler v3?

Here's the specific query:

  const teamArticles = await db.query.articles.findMany({
    where: ownerOrInTeam(articles.userId, authUser), // this is one equality check
    with: {
      pineconeReferences: true,
      outlineDrafts: { // if I do outlineDrafts: true, I am not over 20 anymore
        with: { // this seems to be the edge at which it goes over 20 seemingly
          draftPieces: true
        }
      }
    }
  });

  const teamArticles = await db.query.articles.findMany({
    where: ownerOrInTeam(articles.userId, authUser), // this is one equality check
    with: {
      pineconeReferences: true,
      outlineDrafts: { // if I do outlineDrafts: true, I am not over 20 anymore
        with: { // this seems to be the edge at which it goes over 20 seemingly
          draftPieces: true
        }
      }
    }
  });

This is drizzle syntax by the way if that helps.

divby0•10/3/23, 10:03 AM

Just to make it clear, this happens on several requests, not just this particular query.

kian•10/3/23, 10:24 AM

D1 has

SQLITE_LIMIT_EXPR_DEPTH

SQLITE_LIMIT_EXPR_DEPTH

set to

on the new backend.

kian•10/3/23, 10:24 AM

Your database is likely still on the

alpha

alpha

backend - try running

wrangler d1 info <dbname>

wrangler d1 info <dbname>

kian•10/3/23, 10:25 AM

In any case, if you can get ahold of the actual SQL query that Drizzle generates, you can prefix it with

EXPLAIN QUERY PLAN

EXPLAIN QUERY PLAN

which should tell you the actual expression depth of your query.

divby0•10/3/23, 10:26 AM

Yes, in fact the database was created in alpha backend. 1. How long can I keep this for this existing project, is there any sunset planned? Can I still create new databases with this backend?

Kkian Your database is likely still on the `alpha` backend - try running `wrangler d1 ...

divby0•10/3/23, 10:26 AM

btw for anyone reading this after the fact, the command is

wrangler d1 info <dbname>

wrangler d1 info <dbname>

without the database

kian•10/3/23, 10:27 AM

Not sure on those, cc @Matt

Kkian In any case, if you can get ahold of the actual SQL query that Drizzle generates...

divby0•10/3/23, 10:27 AM

I can try, let me see if I can get it

kian•10/3/23, 10:34 AM

You wouldn't be the first person to run into it, but if you have several real-world examples of queries that hit the limit then it should be reasonable to raise the limit.

divby0•10/3/23, 10:36 AM

Do you speak for cloudflare or do you mean raise it locally for my wrangler v3 setup? It can't be that cloudflare would globally raise the limit because of me, would they?

kian•10/3/23, 10:37 AM

d1-database

kian•10/3/23, 10:37 AM

Just echoing Matt

divby0•10/3/23, 10:39 AM

That makes sense. I am having trouble finding out how exactly to get the executed query to then run it with the explain prefix, but I'll paste it here as soon as I have

kian•10/3/23, 10:39 AM

Looks like you can end it in

toSQL();

toSQL();

https://orm.drizzle.team/docs/goodies#printing-sql-query

divby0•10/3/23, 10:48 AM

Unfortuantely not with the relational query I am using (

db.query.[....]

db.query.[....]

) only with the ones that start differently

GGuillaumeLakano Do you known if this soft limit of 50,000 databases will be increased again late...

Matt Silverlock•10/3/23, 10:49 AM

How many do you need, and is that “now” or “planning for two years from now” - ? We want to allow folks to get increases as needed, but I want to understand if this is an actual blocker for you or not. Keep in mind you can only have ~5k databases per Worker script right now.

GGuillaumeLakano There is a Business plan at $200, but I don't known if WfP is only for Enterpris...

Matt Silverlock•10/3/23, 10:50 AM

If $200/mo is out of your budget - you still have to pay for D1 reads, writes & storage. $200 gets you a LOT of that, but it’s not free!

Ddivby0 Unfortuantely not with the relational query I am using (`db.query.[....]`) only ...

Matt Silverlock•10/3/23, 10:51 AM

Open an issue on GitHub with your query and error. I will not keep up with it here.

MMatt Silverlock Open an issue on GitHub with your query and error. I will not keep up with it he...

divby0•10/3/23, 10:53 AM

Will do

Is there a way to remove my blocker by using wrangler v3 with an alpha / legacy D1 binding? Cause I can't continue development or I'd have to go back to wrangler v2 if it's not possible

MMatt Silverlock How many do you need, and is that “now” or “planning for two years from now” - ?...

GuillaumeLakanoOP•10/3/23, 11:09 AM

We're planning our project release for around Q1 of 2024 too, so we'll wait just after the release of D1 in GA. Before, we've used the DO to have a kind of document DB per user. Was useful but all the eviction problem (sharing memory per Classe) make us move to D1 when you've confirmed to me few weeks/months ago that you'll permit kind of unlimited databases in the future. So, now we're full D1 SQL.

Since, I've read in the blog article « D1: beta open » that finally this kind of unlimited databases will be reserved to your WfP clients (enterprise plan of $2,000 / month), so finally may be we need to have one full classic DB for all the users, or to fallback to our old architecture on DO.

Since yesterday ,you've increased from 25 to 50,000 databases per account, it's a big step, and it's only a soft limit, but we need to understood if we take a risk to add this security layer ( 1 user = 1 D1 database, kind of sandbox ), because if we reach for example 50,000 users, we'll be totally trap in a very bad situation.

GGuillaumeLakano We're planning our project release for around Q1 of 2024 too, so we'll wait just...

Matt Silverlock•10/3/23, 11:17 AM

We will continue to increase it for paid Workers plans (before/at GA). Don’t worry.

MMatt Silverlock We will continue to increase it for paid Workers plans (before/at GA). Don’t wor...

GuillaumeLakanoOP•10/3/23, 11:19 AM

Good to heard, thank you for the answer ! I suppose you can't now spoil a little bit on the future soft limite?

kian•10/3/23, 12:16 PM

https://sqlite.org/draft/releaselog/3_44_0.html

Add support for the JSONB binary representation of JSON, which is both smaller and faster than the corresponding RFC 8529 text format.

Kkian <https://sqlite.org/draft/releaselog/3_44_0.html> > Add support for the JSONB b...

Hello, I’m Allie!•10/3/23, 12:18 PM

Capnp on SQLite wen?

GGuillaumeLakano Good to heard, thank you for the answer ! I suppose you can't now spoil a little...

Matt Silverlock•10/3/23, 1:25 PM

Give me your account ID. You will need a paid Workers subscription though (I will not increase for free accounts). But to be transparent, are you worrying about a limit you are a long, long way from hitting? What limit do we set? No one believes “unlimited”

MMatt Silverlock Give me your account ID. You will need a paid Workers subscription though (I wil...

GuillaumeLakanoOP•10/3/23, 1:51 PM

Thank you! I've just sent you in PM the requested private informations

MMatt Silverlock Open an issue on GitHub with your query and error. I will not keep up with it he...

divby0•10/3/23, 2:12 PM

I couldn't get any info or help on the exact SQL query generated by drizzle yet. But since drizzle is probably the most used ORM for d1, should I just create the issue with the drizzle queries (but not the exact generated SQL)?

Ddivby0 I couldn't get any info or help on the exact SQL query generated by drizzle yet....

Max (@rozenmd)•10/3/23, 2:17 PM

yep

MMax (@rozenmd)yep

divby0•10/3/23, 2:19 PM

Awesome, thanks. Just created it here: https://github.com/cloudflare/workers-sdk/issues/4094

It is not really a bug, so I put it in feature request since that was the only other thing. Feel free to move if not right

nclevenger•10/3/23, 9:02 PM

ugh ... I hate TOML ... does anyone have an example of the proper syntax in wrangler.toml for multiple D1 database bindings?

Unsmart•10/3/23, 9:08 PM

https://developers.cloudflare.com/workers/wrangler/configuration/#d1-databases

[[d1_databases]]
binding = "db1"
database_name = "test-db"
database_id = "c020574a-5623-407b-be0c-cd192bab9545"

[[d1_databases]]
binding = "db2"
database_name = "test-db"
database_id = "c020574a-5623-407b-be0c-cd192bab9545"

[[d1_databases]]
binding = "db1"
database_name = "test-db"
database_id = "c020574a-5623-407b-be0c-cd192bab9545"

[[d1_databases]]
binding = "db2"
database_name = "test-db"
database_id = "c020574a-5623-407b-be0c-cd192bab9545"

Unsmart•10/3/23, 9:09 PM

d1_databases = [
  { binding = "db1", database_name = "db1", database_id = "1" },
  { binding = "db2", database_name = "db2", database_id = "2" }
]

d1_databases = [
  { binding = "db1", database_name = "db1", database_id = "1" },
  { binding = "db2", database_name = "db2", database_id = "2" }
]

UUnsmart or ```toml d1_databases = [ { binding = "db1", database_name = "db1", databas...

Dahoom•10/3/23, 9:10 PM

thats .toml? or is there a way to specify the bindings in js

Unsmart•10/3/23, 9:10 PM

thats .toml not valid js dont think you can use js but there might be an option for json

Dahoom•10/3/23, 9:11 PM

interesting, never knew you know format it that way

Nnclevenger ugh ... I hate TOML ... does anyone have an example of the proper syntax in wran...

Hello, I’m Allie!•10/4/23, 7:56 AM

Hey Nathan, try

wrangler -j

wrangler -j

SPOILER_wrangler.json288B

Old Man Maker•10/4/23, 10:07 AM

Hi there.
This causes and error:

const batcher = await db.prepare("SELECT * FROM table WHERE ref = ?1");

const batcher = await db.prepare("SELECT * FROM table WHERE ref = ?1");

this does not:

const batcher = await db.prepare("SELECT * FROM table WHERE ref = ?");

const batcher = await db.prepare("SELECT * FROM table WHERE ref = ?");

The inclusion of an indexer should be optional enough not to cause an issue.
Not sure if it's raised somewhere as a consideration?

Vikash•10/4/23, 11:30 AM

Any way to improve performance, it seems all queries takes 1.xx seconds, even on Cloudflare dashboard simple products table with 5 rows -

VVikash Any way to improve performance, it seems all queries takes 1.xx seconds, even on...

Isaac McFadyen•10/4/23, 11:34 AM

If all queries are slow can you check if you're on the beta or alpha version, and also whether anything else is slow other than D1? (for example, D1 is slow on my internet if I have an ongoing backup, since it saturates my internet link and slows everything else down)

VVikash Any way to improve performance, it seems all queries takes 1.xx seconds, even on...

Max (@rozenmd)•10/4/23, 11:41 AM

the speed of the API in the dashboard isn't indicative of what you should be seeing from your Worker - deploy a Worker and see how long your requests are taking

IIsaac McFadyen If all queries are slow can you check if you're on the beta or alpha version, an...

Vikash•10/4/23, 11:46 AM

Only D1 is slow, I am using other options like KV which works fine. How to check alpha or beta? I created the database via dashboard 2 weeks ago.

VVikash Only D1 is slow, I am using other options like KV which works fine. How to check...

Isaac McFadyen•10/4/23, 11:47 AM

If you created it that recently it should be

beta

beta

, yes.

Isaac McFadyen•10/4/23, 11:47 AM

Alpha is the older D1 engine that doesn't work as well and is deprecated.

MMax (@rozenmd)the speed of the API in the dashboard isn't indicative of what you should be see...

Vikash•10/4/23, 11:47 AM

I deployed the worker and getting same result -

Vikash•10/4/23, 11:49 AM

While the KV is returning in less then 50ms

const teamArticles = await db.query.articles.findMany({ where: ownerOrInTeam(articles.userId, authUser), // this is one equality check with: { pineconeReferences: true, outlineDrafts: { // if I do outlineDrafts: true, I am not over 20 anymore with: { // this seems to be the edge at which it goes over 20 seemingly draftPieces: true } } } });

[[d1_databases]] binding = "db1" database_name = "test-db" database_id = "c020574a-5623-407b-be0c-cd192bab9545" [[d1_databases]] binding = "db2" database_name = "test-db" database_id = "c020574a-5623-407b-be0c-cd192bab9545"

Do you known if this soft limit of 50,000 databases will be increased again later for the paid plans

Similar Threads

Similar Threads

Similar Threads