Cloudflare Developers•2y ago

nice. i assume we expect the 10gb or 50k db limit to increase sooner than later

FFrank nice. i assume we expect the 10gb or 50k db limit to increase sooner than later

Chaika•5/1/24, 9:11 PM

The DB limit is soft, the size per db limit is hard

Chaika•5/1/24, 9:12 PM

d1-database

Chaika•5/1/24, 9:12 PM

d1-database

Chaika•5/1/24, 9:13 PM

they've already said they can support virtually limitless (millions) of separate dbs

CChaika they've already said they can support virtually limitless (millions) of separate...

FrankOP•5/1/24, 9:15 PM

awesome. thanks. just waiting on dynamic binding then.
and be careful about limit bindings per worker at a given time. I think ~5k right now

FrankOP•5/1/24, 9:16 PM

hopefully bindings can be both created and released dynanmically

FFrank awesome. thanks. just waiting on dynamic binding then. and be careful about limi...

Chaika•5/1/24, 9:26 PM

when I think about dynamic bindings I think about Worker for Platform's Dynamic Dispatch binding, where it's just a singular binding you can use to get access to a worker/instance, ex:

    const worker = env.DISPATCHER.get("customer-worker-1");
    var responseFromUserScript = worker.fetch(req);

    const worker = env.DISPATCHER.get("customer-worker-1");
    var responseFromUserScript = worker.fetch(req);

Chaika•5/1/24, 9:27 PM

I don't think they've said much about dynamic bindings, but I imagine it'd be something like that where you've got maybe an entry binding (or maybe no entry binding at all and it's just part of the runtime/env object, all dynamic), but eitherway each binding wouldn't become part of your env or stick around, except for during that request

FrankOP•5/1/24, 9:32 PM

any idea how migrations work when u have so many dbs

Rubi•5/2/24, 12:38 AM

basically a Go script to execute .sql concurrently on d1 rest api

Kkian `wrangler d1 export`

Softbear•5/2/24, 5:02 AM

Missed that news

CChaika when I think about dynamic bindings I think about Worker for Platform's Dynamic ...

Beny•5/2/24, 7:12 AM

I created something like this with Worker for Platforms. It uploads a new script with the binding that is essentially a D1 proxy over RPC. The only downside is that it can't be used immediately because the dispatcher doesn't think the script is valid straight away. The only solution I could get to work was implementing a delay of a few seconds before the script is usable. Apart from that it works very well

Sam•5/2/24, 8:12 PM

Hey all I wanted to echo https://community.cloudflare.com/t/different-errors-on-d1-randomly-happening-once-in-a-while/599421 -- Every few days now I get a "D1_ERROR: D1 storage operation exceeded timeout which caused object to be reset."

Is this a known issue? Very limited results on google & none in Discord

(

fdb7837214fdd80e018b0bf3ae6355c9

fdb7837214fdd80e018b0bf3ae6355c9

<redacted>-nx-production

<redacted>-nx-production

)

malone•5/2/24, 9:11 PM

I'm a bit confused with the upgrade process. When i ran wrangler d1 info, it said 'beta' instead of alpha and I didn't do anything yet.

malone•5/2/24, 9:11 PM

When i updated wrangler the version has been removed from the table entirely

malone•5/2/24, 9:11 PM

malone•5/2/24, 9:12 PM

with a brand new db i just created it also says beta

Unsmart•5/2/24, 9:13 PM

an alpha db would say alpha

Unsmart•5/2/24, 9:13 PM

beta is the most up to date version

malone•5/2/24, 9:13 PM

ok cool! so nothing to worry about. thanks @Unsmart | Tech debt

ALECKS•5/3/24, 12:26 AM

Is it possible to connect to a D1 database programmatically, from a remote host? For ex, I want to run a js script on my machine that connects to a D1 instance and does some stuff. I want to do this with JS, instead of .sql files, which I know I can run through wrangler d1 execute

Original message was deleted

forslunds•5/3/24, 3:05 AM

The local is a sqlite file inside wrangler folder. Open it in a sqlite program and export

stibbs•5/3/24, 3:35 AM

Apologies if this has been asked before, but is there a way to run down migrations? Or is it advised to:
1) create a new up migration to unwind a change [I am assuming this is the answer]; or
2) revert to prior state via time travel?

FrankOP•5/3/24, 4:59 AM

https://alexgarcia.xyz/blog/2024/building-new-vector-search-sqlite/index.html
pretty cool if this works well and eventually lands in D1 land

I'm writing a new vector search SQLite Extension

sqlite-vec is an new vector search SQLite extension, coming soon!

Kysan•5/3/24, 6:28 PM

https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database
what is the rate limit for the http api endpoint ?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Kysan•5/3/24, 6:29 PM

1200 requests per 5 minutes ty

KKysan 1200 requests per 5 minutes ty

Chaika•5/3/24, 6:32 PM

yea and that's shared among all endpoints/dashboard/etc for your user, some endpoints like purge by url have special ratelimits as well, but d1 doesn't

Chaika•5/3/24, 6:33 PM

?d1-remote-connections

Flare•5/3/24, 6:33 PM

Accessing D1 from outside workers:

D1 is based on SQLite, which does not natively support remote connections and isn't MySQL/PostgreSQL wire compatible, nor does D1 itself offer that functionality. As such, your options are:

1 -> The Cloudflare API Query Endpoint (https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database), with the upside of needing less setup, all requests have to go back to Cloudflare's core, so request latency is far from optimal from some locations. You would also be subject to the API Rate limit of 1200 requests per 5 minutes.

2 -> Create a worker to proxy requests to your D1 Database. Either tightly bound to your application/schema or loose and just accepting any query. This would allow your worker to execute potentially closer to your D1 Database and avoid the api rate limit.

Miyagi•5/3/24, 8:18 PM

Hello, I have had success with the D1 in development all the way until production deployment. I have bound it using the updated syntax from docs (Pages). When i attempt to query when deployed, It crashes with Application Error.

Works just fine in wrangler preview.

AccountId: 77e1b1d919d86d253e4967517aab4437

Steved•5/3/24, 10:11 PM

Does anyone have an example of storing binary data in D1 using Drizzle?
Drizzle seems to be expecting a 'Buffer' type, but when I pass one in I'm getting an assert in Miniflare where it isn't expecting a Buffer in ArrayBufferView (ALLOWED_ARRAY_BUFFER_VIEW_CONSTRUCTORS). It's expecting one of the typed array names. At this point I can't tell whether the issue is in Drizzle or Miniflare (or devalue package).

Steved•5/3/24, 10:19 PM

If I ignore types and just pass a Uint8Array through it works.

skip•5/3/24, 10:51 PM

I am having trouble running my migrations on Prod. Works file locally, and I can manually copy and paste them into the console fine. But when I try "wrangler d1 migrations apply xxx" I get "No migrations to apply!" even though it's a fresh database.

I can see there is a table "sqlite_schema". Does this track what migrations have been applied so far?

I thought I'd try to rename that table, and see if that resets my migration point to see if it starts fresh - but I get an error "table sqlite_master may not be modified"

Sskip 👋 I am having trouble running my migrations on Prod. Works file locally, and I...

Isaac McFadyen•5/3/24, 11:09 PM

sqlite_schema is unrelated to migrations, it's an internal table SQLite uses for various things.

IIsaac McFadyen sqlite_schema is unrelated to migrations, it's an internal table SQLite uses for...

skip•5/3/24, 11:16 PM

OK thanks... would you know how migrations decide what migration point has or has not run yet?

Because ATM my database it totally empty and it thinks none of the migrations need to run

Isaac McFadyen•5/3/24, 11:17 PM

I'm not sure, sorry - someone else would probably know the answer.

stibbs•5/4/24, 4:46 AM

How do I use sqlite extensions with D1? E.g.,

uuid.c

uuid.c

(https://sqlite.org/src/file/ext/misc/uuid.c) to generate UUIDs

EDIT: Or should I be creating the UUID's myself prior to interacting with D1?

kian•5/4/24, 5:16 AM

If the extension isn't already enabled in D1 then you can't use it - so you'll need to make the UUIDs yourself, i.e with crypto.randomUUID()

Bilal B•5/4/24, 6:15 AM

Hey I'm trying to run SQL commands from

console

console

tab in the interface but it says I'm not authorized

Bilal B•5/4/24, 6:15 AM

I'm the owner of the account

Bilal B•5/4/24, 6:16 AM

never mind, I was using

BEGIN... COMMIT;

BEGIN... COMMIT;

which I guess is not supported

scruffy•5/4/24, 1:57 PM

I have two workers (in

a/

a/

and

b/

b/

), both with D1 bindings to

DB

DB

in their

wrangler.toml

wrangler.toml

s. I created the D1 inside

a/

a/

, added some data, and can retrieve it using

pnpm wrangler d1 execute DB --local --command="SELECT * FROM users"

pnpm wrangler d1 execute DB --local --command="SELECT * FROM users"

inside

a/

a/

. Ideally, I'd like to run the same command in

b/

b/

, but I'm getting an error:

no such table: users

no such table: users

. I'm guessing this is because the local D1 instance is inside

a/.wrangler/state/v3/d1/miniflare-D1DatabaseObject

a/.wrangler/state/v3/d1/miniflare-D1DatabaseObject

...is there any way to make this (or similar) work?

scruffy•5/4/24, 2:00 PM

(am trying to boil down a problem I'm facing in a monorepo)

scruffy•5/4/24, 4:25 PM

For anyone running into this, I solved it by using Miniflare for running the workers and setting

mf.options.d1Persist: 'data-worker/.wrangler/state/v3/d1

mf.options.d1Persist: 'data-worker/.wrangler/state/v3/d1

, i.e. the path which contains

miniflare-D1DatabaseObject

miniflare-D1DatabaseObject

inside the worker where D1 was created

scruffy•5/4/24, 4:25 PM

(in this snippet, the relevant worker is called

data-worker

data-worker

)

Fabian•5/4/24, 5:53 PM

I have seen many use Drizzle here - is there any way to prepare queries? I have a rather "complex" query that when running locally it has basically no impact on runtime, but when deployed it takes 10-20ms to compile. Idk if the "CPUs" on Workers are really that trash or if I am doing something wrong. (side note, it's inside a batch, so no, it's not latency related - upon removing it from the batch performance gets instantly better. Also assured it is query compilation by pasting the query multiple times in the code without executing it)

Kkian If the extension isn't already enabled in D1 then you can't use it - so you'll n...

skip•5/5/24, 12:06 AM

Is there a list of enabled extensions?

nice. i assume we expect the 10gb or 50k db limit to increase sooner than later

Similar Threads

Similar Threads

Similar Threads