Cloudflare Developers•2y ago

what does your table list look like?

MyZeD•4/10/24, 7:21 PM

got it....

MMax (@rozenmd)what does your table list look like?

MyZeD•4/10/24, 7:21 PM

was the foreign key: "FOREIGN KEY (price) REFERENCES price (price)" sorry

MyZeD•4/10/24, 7:22 PM

I'm sorry for that

Max (@rozenmd)OP•4/10/24, 7:23 PM

all good

unk•4/11/24, 5:11 AM

Hi.
Why do I get this error?

Error: D1_ERROR: too many SQL variables at offset 484

Error: D1_ERROR: too many SQL variables at offset 484

I use drizzle.

const data = await db.query.projects.findMany({
    limit: PAGE_LIMIT,
    where: inArray(
    girls.id,
    girlsIds.map(({ id }) => id),
    ),
    offset: calcOffset(Number(qs.page), PAGE_LIMIT),
})

const data = await db.query.projects.findMany({
    limit: PAGE_LIMIT,
    where: inArray(
    girls.id,
    girlsIds.map(({ id }) => id),
    ),
    offset: calcOffset(Number(qs.page), PAGE_LIMIT),
})

Max (@rozenmd)OP•4/11/24, 9:06 AM

(I am probably going to be posting this every few days until we shut down the old alpha DBs)

On the off-chance you're still using an alpha D1 database, we've made a guide to make it easier to move across to the newer, faster (40x more requests per second) D1 DBs:

https://developers.cloudflare.com/d1/platform/alpha-migration

Original message was deleted

johtso•4/11/24, 9:25 AM

No, you're all good if you use batch, not 100% what the limits are for a single batch operation, but I think they're really high.. and not based on params

Gyurmatag•4/11/24, 9:44 AM

Hi guys! If I want to move from D1 to another Cloud provider, then what are my options? Thank you very much for the answers!

GGyurmatag Hi guys! If I want to move from D1 to another Cloud provider, then what are my o...

Beny•4/11/24, 9:49 AM

Export your data [1]:

wrangler d1 export <DATABASE_NAME> --output db.sql --remote

wrangler d1 export <DATABASE_NAME> --output db.sql --remote

Then upload it to another cloud & either use their API or Hyperdrive to connect your worker to the db [2]
[1] https://developers.cloudflare.com/workers/wrangler/commands/#export
[2] https://developers.cloudflare.com/hyperdrive/

CCole Xemi Maximum bound parameters per query is 100. https://developers.cloudflare.com/d1/...

unk•4/11/24, 9:57 AM

Suppose you wrote a query like this using offset

const data = await db.query.projects.findMany({
  where: inArray(
    projects.key,
    [... //over 100 items]
  ),
  offset: 50,
})

const data = await db.query.projects.findMany({
  where: inArray(
    projects.key,
    [... //over 100 items]
  ),
  offset: 50,
})

but But this exceeds the limit.
Is it not feasible to combine the search and OFFSET for more than 100 arrays?

Gunnar•4/11/24, 10:28 AM

Hi, sorry if this has been asked before, but how are you guys gonna work with this limit, if your service explodes in popularity?
Maximum database size 10 GB (Workers Paid)
https://developers.cloudflare.com/d1/platform/limits/

Cloudflare Docs

Limits · Cloudflare D1 docs

If you would like to explore other storage solutions for your application, Cloudflare also offers Workers KV, Durable Objects, and R2.

GGunnar Hi, sorry if this has been asked before, but how are you guys gonna work with th...

bun•4/11/24, 10:38 AM

10gb is plenty enough for a lot of users

bun•4/11/24, 10:38 AM

They recommend using multiple databases in any way

bun•4/11/24, 10:39 AM

Like theres this possibility of having a db per customer even

GGunnar Hi, sorry if this has been asked before, but how are you guys gonna work with th...

Max (@rozenmd)OP•4/11/24, 10:44 AM

(my canned reply to this is):

D1 is built to work to the strengths of Cloudflare's network, so we can't offer extremely large databases while also loading them quickly. It's significantly easier for us to offer millions of DBs with a size limit instead.

D1 will require a change in how you think about your project's data if you're looking to build a big platform. Instead of storing all of your data in a single DB, you would store each customer's data in their own DB (and we're building tooling to make it easier to dynamically access multiple DBs in a single Worker without redeploying)

MMax (@rozenmd)(my canned reply to this is): D1 is built to work to the strengths of Cloudflar...

Hello, I’m Allie!•4/11/24, 10:48 AM

One thing I've been wondering, if the main issue to having large databases is startup speed, what effect does a large database have on execution speed? As in, if you could somehow assume that the DB is always hot, would the upper limit for storage be much higher?

MMax (@rozenmd)(my canned reply to this is): D1 is built to work to the strengths of Cloudflar...

forslunds•4/11/24, 10:51 AM

in a single worker.. not a single pages function?

Fforslunds in a single worker.. not a single pages function? 🥲

Hello, I’m Allie!•4/11/24, 10:52 AM

All of the Functions in a Pages Project are bundled into a Worker, so it should apply to Functions too

HHello, I’m Allie!All of the Functions in a Pages Project are bundled into a Worker, so it should ...

forslunds•4/11/24, 10:55 AM

if i got a cent for every time i heard that..

HHello, I’m Allie!All of the Functions in a Pages Project are bundled into a Worker, so it should ...

Max (@rozenmd)OP•4/11/24, 10:59 AM

this, but I always worked around Pages functions not having all the features by using a separate Worker for my API

Workers and Pages becoming the same "thing" will make this better

GGunnar Hi, sorry if this has been asked before, but how are you guys gonna work with th...

Beny•4/11/24, 11:03 AM

If your service gets to that size then you shouldn't be using sqlite (which is what D1 is) anyway.
I have one app with 8000+ users and over 11 million records in the db and it is under 2gb, so I dont think it's an issue unless you are storing some big data in there

MMax (@rozenmd)(my canned reply to this is): D1 is built to work to the strengths of Cloudflar...

Gunnar•4/11/24, 11:04 AM

Is there a max db count limit?

GGunnar Is there a max db count limit?

Beny•4/11/24, 11:05 AM

It's 50,000 but they've mentioned they've just set that so people don't go crazy. It can be increased to anything.
https://developers.cloudflare.com/d1/platform/limits/

Cloudflare Docs

Limits · Cloudflare D1 docs

If you would like to explore other storage solutions for your application, Cloudflare also offers Workers KV, Durable Objects, and R2.

BBeny It's 50,000 but they've mentioned they've just set that so people don't go crazy...

Gunnar•4/11/24, 11:07 AM

Oh jesus, sorry, I'm blind, it's the top line

GGunnar Is there a max db count limit?

Max (@rozenmd)OP•4/11/24, 11:08 AM

when we release dynamic bindings, it'll look a bit more like DO's limits page

Beny•4/11/24, 11:09 AM

Next we need an unlimited KV's limit

That one is pretty small

BBeny Next we need an unlimited KV's limit 😄 That one is pretty small

Gunnar•4/11/24, 11:11 AM

There is a "Limit Increase Request Form" for KV
https://developers.cloudflare.com/kv/platform/limits/

Cloudflare Docs

Limits · Cloudflare Workers KV

Request adjustments to limits that conflict with your project goals by contacting Cloudflare. To request an increase to a limit, complete the Limit …

BBeny Next we need an unlimited KV's limit 😄 That one is pretty small

Hello, I’m Allie!•4/11/24, 11:12 AM

You can also just create new “namespaces” by giving each a prefix

HHello, I’m Allie!You can also just create new “namespaces” by giving each a prefix

Beny•4/11/24, 11:14 AM

Yeah this is a thing, but that workaround wouldn't be good in combination with Workers for Platforms, otherwise the user workers will be able to see other's data

GGunnar There is a "Limit Increase Request Form" for KV https://developers.cloudflare.co...

Beny•4/11/24, 11:18 AM

I've never tried the limit increase before but I've seen some people here mention their requests went unanswered for weeks so it puts me off a bit (and it is strange hosting it on a Google form, doesn't feel official)