Hey Cloudflare team, I saw that the current limit is 10GB per database, but we need to store data f

Hey Cloudflare team,

I saw that the current limit is 10GB per database, but we need to store data for about 200 million users, which would require something like 200GB. With the 10GB limit, we can only fit data for around 10-20 million users.

Are there any plans to increase the limit, or is there any way to upgrade to larger storage? Just trying to figure out the best approach for our project.

Thanks in advance!

Xxiaolong01 Hey Cloudflare team, I saw that the current limit is 10GB per database, but we ...

Max (@rozenmd)•9/5/24, 5:29 PM

If you're set on storing that data in a single database, we'd recommend using Hyperdrive instead

Max (@rozenmd)•9/5/24, 7:42 PM

D1 is built around Cloudflare's network, which is really efficient at loading lots of small files really quickly - we'd offer bigger DBs but the cold starts would be brutal

GGustavo Do you think it's *technically possible* to raise the limit to hundreds of GBs i...

Max (@rozenmd)•9/5/24, 7:46 PM

Unlikely, Hyperdrive is a much better fit for that

Max (@rozenmd)•9/5/24, 7:47 PM

With the 50k DBs D1 starts with, we're also building a means of dynamic binding as many DBs as you want (and related tooling) to make it easier to manage, but that's a new way of building software that folks might not be ready for

Gagan Suie•9/6/24, 3:57 AM

is there anything wrong with code like this?

/**
 * Update milestone by milestoneId
 * @param request
 * @param env
 * @returns
 */
const updateMilestone = async (request: Request, env: any) => {
    try {
        const params = new URL(request.url).searchParams
        const milestoneId = parseInt(params.get('milestoneId') || '')
        const milestoneData: any = await request.json()
        const fields: string[] = []
        const values: any[] = []
        let index = 1
        for (const [key, value] of Object.entries(milestoneData)) {
            if (value !== undefined) {
                fields.push(`${key} = ?${index}`)
                values.push(value)
                index++
            }
        }
        if (fields.length === 0) {
            return new Response(JSON.stringify({ error: 'No fields to update' }), { status: 400 })
        }
        values.push(milestoneId)
        const query = `UPDATE milestone SET ${fields.join(', ')}, updatedAt = CURRENT_TIMESTAMP WHERE _id = ?${index}`
        const { success } = await env.DB.prepare(query)
            .bind(...values)
            .run()
        return new Response(JSON.stringify({ success }), { status: 200 })
    } catch (err: any) {
        return new Response(JSON.stringify(err), { status: 500 })
    }
}

/**
 * Update milestone by milestoneId
 * @param request
 * @param env
 * @returns
 */
const updateMilestone = async (request: Request, env: any) => {
    try {
        const params = new URL(request.url).searchParams
        const milestoneId = parseInt(params.get('milestoneId') || '')
        const milestoneData: any = await request.json()
        const fields: string[] = []
        const values: any[] = []
        let index = 1
        for (const [key, value] of Object.entries(milestoneData)) {
            if (value !== undefined) {
                fields.push(`${key} = ?${index}`)
                values.push(value)
                index++
            }
        }
        if (fields.length === 0) {
            return new Response(JSON.stringify({ error: 'No fields to update' }), { status: 400 })
        }
        values.push(milestoneId)
        const query = `UPDATE milestone SET ${fields.join(', ')}, updatedAt = CURRENT_TIMESTAMP WHERE _id = ?${index}`
        const { success } = await env.DB.prepare(query)
            .bind(...values)
            .run()
        return new Response(JSON.stringify({ success }), { status: 200 })
    } catch (err: any) {
        return new Response(JSON.stringify(err), { status: 500 })
    }
}

Gagan Suie•9/6/24, 3:58 AM

im specifically referring to the update query

GGagan Suie is there anything wrong with code like this? ```/** * Update milestone by mile...

Max (@rozenmd)•9/6/24, 4:53 AM

apart from the SQL injection?

Gagan Suie•9/6/24, 4:54 AM

Yes that's what I was worried about. Is there a better approach to writing something like this?

Max (@rozenmd)•9/6/24, 5:01 AM

nothing that comes to mind (its 7am here, brain still booting) other than strictly validating that the inbound fields are what you expect via zod (+ it integrates with hono: https://hono.dev/docs/guides/validation#zod-validator-middleware)

Validation - Hono

Ultrafast web framework for Cloudflare Workers, Fastly Compute, Deno, Bun, Vercel, Node.js, and others. Fast, but not only fast.

MMax (@rozenmd)nothing that comes to mind (its 7am here, brain still booting) other than strict...

Gagan Suie•9/6/24, 5:23 AM

We use @cloudflare/itty-router-openapi

神神龙出海 Click to see attachment

神

神龙出海•9/6/24, 6:02 AM

@Max (@rozenmd) Brother, I have this problem again in my database

神

神龙出海•9/6/24, 6:03 AM

Please help me check the problem, which directly causes my product to be unavailable

Rubi•9/6/24, 6:11 AM

hi umm, I got an error when exporting D1 database

✘ [ERROR] near "limit": syntax error at offset 40: SQLITE_ERROR

✘ [ERROR] near "limit": syntax error at offset 40: SQLITE_ERROR

Rubi•9/6/24, 6:11 AM

command:

npx wrangler d1 export db-name --remote --output ./database.sql

npx wrangler d1 export db-name --remote --output ./database.sql

神神龙出海 Please help me check the problem, which directly causes my product to be unavail...

Max (@rozenmd)•9/6/24, 7:33 AM

Are you seeing intermittent errors or is it completely unavailable?

神

神龙出海•9/6/24, 7:34 AM

There have been a lot of intermittent errors in the past few days, and they went away after a while just now.

神神龙出海 There have been a lot of intermittent errors in the past few days, and they went...

Max (@rozenmd)•9/6/24, 7:34 AM

I'll have the team investigate deeper when they get online (US hours)

神

神龙出海•9/6/24, 7:35 AM

Okay, please solve this problem completely as soon as possible, otherwise D1 will be very unreliable as a production environment database.

petar•9/6/24, 10:43 AM

Hi folks. My team and I are starting to use D1 where we need to have one simple table and do some simple checks from a CF worker. The way we want to populate the database (regularrly, every two weeks or so) is from a Rails app, where the app downloads a CSV file, process the data and insert/update the D1 table. Pretty straight forward, but the issue we are facing is with the CF API as it's not clear how to insert the new data and populate the table.
In the POST body there are 3 possible objects that we can pass: https://developers.cloudflare.com/api/operations/cloudflare-d1-import-database but it's not clear what we should do next, because the response always have an "upload_url" which is not clear if we need to use that URL for something or not. Is there an example, even better if it's in Ruby, so we can test it? Or a written SOP or something of how should the whole process go.

Thanks in advance!

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

matthew•9/6/24, 12:38 PM

Do you have to use the REST API? Might be easier to use the D1 API in Workers directly.

matthew•9/6/24, 1:07 PM

Question: can I get raw results when using D1 batch()? It seems there is no way of avoiding the cost of turning everything into JSON, if I want the perf benefits of batching queries.

Mmatthew Question: can I get raw results when using D1 batch()? It seems there is no way ...

Hard@Work•9/6/24, 1:08 PM

Not sure I understand what you mean by "raw results"?

HHard@Work Not sure I understand what you mean by "raw results"?

matthew•9/6/24, 1:10 PM

Results in array format, e.g. https://developers.cloudflare.com/d1/build-with-d1/d1-client-api/#await-stmtraw

Cloudflare Docs

Query D1 | Cloudflare D1 docs

D1 is compatible with most SQLite’s SQL convention since it leverages SQLite’s query engine. D1 client API allows you to interact with a D1 database from within a Worker.

Mmatthew Results in array format, e.g. https://developers.cloudflare.com/d1/build-with-d1...

Hard@Work•9/6/24, 1:41 PM

I mean, I don't think there is actually that much of a difference, but no, afaik there isn't a way to do batch raw queries atm

Original message was deleted

Hard@Work•9/6/24, 1:49 PM

Sorry, I meant specifically for performance, but good point

Rubi•9/7/24, 4:49 PM

hi, I can't export my database

and there is no other way than using wrangler export

Rubi•9/7/24, 4:57 PM

oh

turns out I was having a column name called

limit

limit

and it's now working after I rename the table

https://github.com/cloudflare/workers-sdk/issues/6599

ac•9/7/24, 6:03 PM

On a related note ^, do you all ever intend to allow exporting D1 as a simple sqlite file, rather than the complicated wrangler export (which seems pretty prone to error)? This would make local development so much easier.

Original message was deleted

Cyb3r-Jak3•9/8/24, 3:10 AM

I mean step one would be building a worker to handle the queries from Grafana. Then you could do it with a Grafana REST plugin

Original message was deleted

Cyb3r-Jak3•9/8/24, 3:11 AM

There is. The only thing is it is rate limited

Cyb3r-Jak3•9/8/24, 3:13 AM

Yeah you can use a REST API connetion in Grafana to query it

JJacob drizzle.config.ts ```ts import { defineConfig } from "drizzle-kit"; import fs fr...

Iberius•9/8/24, 1:27 PM

btw i got D1 and drizzle working if you ened help lmk

Shravan•9/8/24, 5:28 PM

Blobs seem to be returning a number[] instead of arraybuffer in d1. Does anyone know why? there's a few old threads about this

Shravan•9/8/24, 6:36 PM

oh the tdocs seem to say

2024-09-08.0936.Brave_Browser.Query_D1_Cloudflare_D1_docs.png

Shravan•9/8/24, 6:39 PM

ah so, the return types are always js objects? so blob wil be an array of number?

Shravan•9/8/24, 6:51 PM

thanks, i was going i little crazy trying to figure out what was going in

Kyrre•9/9/24, 8:34 AM

Hey! Why does one D1 database consistently have ~ +80 ms query time compared to another, when querying an identical, empty table? Both run in EEUR, and both are the "production" version. None have any load.

Spuckwaffel•9/9/24, 8:59 AM

Hey there, i was wondering why when you create a Table on the CF dashboard, you cannot have multiple columns with a primary key. Is this intentional?
If you create the DB using the Console you can specify something like CREATE TABLE ... (APIKey TEXT, username TEXT, ... , PRIMARY KEY (APIKey, username) );CREATE TABLE ... (APIKey TEXT, username TEXT, ... , PRIMARY KEY (APIKey, username) ); which executes successfully.

Original message was deleted

Kyrre•9/9/24, 9:28 AM

Thanks! Any way to determine this? Or set more fine-grained location?

Original message was deleted

Kyrre•9/9/24, 9:40 AM

So strange, when the main selling point is being close to the user. Guess we have to start migrating away from D1.

IEatBeans•9/10/24, 1:35 AM

Is it possible to create an api token with access to a specific d1 db, instead of just all of them on my account?

IIEatBeans Is it possible to create an api token with access to a specific d1 db, instead o...

Cyb3r-Jak3•9/10/24, 3:09 AM

It is not currently, and I don't believe it is currently on the road map

MMax (@rozenmd)(my canned reply to this is): D1 is built to work to the strengths of Cloudflar...

�

𝓈𝒽𝓎•9/10/24, 5:59 AM

hello @Max (@rozenmd) I'm currently building my startup which would be a POS saas and I thought a database per business would be great, I saw you 5 months ago mentioning a tooling that will help with this process that will be out in couple of weeks/months, idk what to do now, is there a clunky workflow you recommend me to start with for now?

oof2win2•9/10/24, 1:52 PM

wait for birthday week which is in 2 weeks

oof2win2•9/10/24, 1:52 PM

if they have something big like that (would probably be for queues + kv and other products) then theyll probably announce it then

/** * Update milestone by milestoneId * @param request * @param env * @returns */ const updateMilestone = async (request: Request, env: any) => { try { const params = new URL(request.url).searchParams const milestoneId = parseInt(params.get('milestoneId') || '') const milestoneData: any = await request.json() const fields: string[] = [] const values: any[] = [] let index = 1 for (const [key, value] of Object.entries(milestoneData)) { if (value !== undefined) { fields.push(`${key} = ?${index}`) values.push(value) index++ } } if (fields.length === 0) { return new Response(JSON.stringify({ error: 'No fields to update' }), { status: 400 }) } values.push(milestoneId) const query = `UPDATE milestone SET ${fields.join(', ')}, updatedAt = CURRENT_TIMESTAMP WHERE _id = ?${index}` const { success } = await env.DB.prepare(query) .bind(...values) .run() return new Response(JSON.stringify({ success }), { status: 200 }) } catch (err: any) { return new Response(JSON.stringify(err), { status: 500 }) } }

Hey Cloudflare team, I saw that the current limit is 10GB per database, but we need to store data f

Similar Threads