Cloudflare Developers•2y ago

its basically an sqlite file, maybe you need to do some load test

RRubi its basically an sqlite file, maybe you need to do some load test

Jordan•5/1/24, 11:17 AM

I know it uses SQLite, but D1 is an actual API in-front of that

JJordan I assume this is done with the local version of D1? If that's true is the local ...

Hello, I’m Allie!•5/1/24, 11:24 AM

The D1 HTTP API is not available anywhere other than in “prod”

Hello, I’m Allie!•5/1/24, 11:25 AM

You could probably build a fake one with a Worker

Hello, I’m Allie!•5/1/24, 11:25 AM

But then you might as well use another SQLite provider

kian•5/1/24, 11:28 AM

Yeah, I'd just use normal SQLite if you're not using it for just local development

Jordan•5/1/24, 11:35 AM

Roger

Softbear•5/1/24, 3:55 PM

Hello in the docs for backups there is this line
"Time Travel does not yet allow you to clone or fork an existing database to a new copy. In the future, Time Travel will allow you to fork (clone) an existing database into a new database, or overwrite an existing database." this makes the restore way less useful than I imagined, I can't event restore a single table without loosing all my state.

https://developers.cloudflare.com/d1/reference/time-travel/

Do you plan to implement this in the near future ? And what about the export to sql or .sqlite, I read that you planned to reitroduce it, but still no news.

Cloudflare Docs

Time Travel and backups · Cloudflare D1 docs

Time Travel is D1’s approach to backups and point-in-time-recovery, and allows you to restore a database to any minute within the last 30 days.

kian•5/1/24, 4:03 PM

wrangler d1 export

wrangler d1 export

kian•5/1/24, 4:03 PM

https://developers.cloudflare.com/d1/build-with-d1/import-data/#export-an-existing-d1-database

ajgeiss0702•5/1/24, 4:22 PM

Was I imagining things or did the blog post talking about D1 replication and sessions used to say that the API is available today and will use the old model and will automatically upgrade when replication is released?

I thought i read that when it was released, and sat down to look into it now, and I dont see it. It's very possible I just misread it the first time

kian•5/1/24, 4:56 PM

https://blog.cloudflare.com/building-d1-a-global-database is the blog you're thinking of but the Sessions API isn't a thing yet & there are no plans to make the existing API behave differently.

Since the Sessions API changes D1's consistency model, developers must opt-in to the new API. Existing D1 API methods are unchanged and will still have the same snapshot isolation consistency model as before. However, only queries made using the new Sessions API will use replicas.

ajgeiss0702•5/1/24, 5:04 PM

Yeah, I was searching through that again. I could have sworn it said something about how you can write code for sessions today and theyll fall back to the old behaviour, then automatically use the new behaviour once its available (and the old api would remain the same)

Vlady•5/1/24, 7:27 PM

Here goes nothing

VVlady Here goes nothing 👀 🤞🏼

Frank•5/1/24, 9:09 PM

nice. i assume we expect the 10gb or 50k db limit to increase sooner than later

FFrank nice. i assume we expect the 10gb or 50k db limit to increase sooner than later

Chaika•5/1/24, 9:11 PM

The DB limit is soft, the size per db limit is hard

Chaika•5/1/24, 9:12 PM

d1-database

Chaika•5/1/24, 9:12 PM

d1-database

Chaika•5/1/24, 9:13 PM

they've already said they can support virtually limitless (millions) of separate dbs

CChaika they've already said they can support virtually limitless (millions) of separate...

Frank•5/1/24, 9:15 PM

awesome. thanks. just waiting on dynamic binding then.
and be careful about limit bindings per worker at a given time. I think ~5k right now

Frank•5/1/24, 9:16 PM

hopefully bindings can be both created and released dynanmically

FFrank awesome. thanks. just waiting on dynamic binding then. and be careful about limi...

Chaika•5/1/24, 9:26 PM

when I think about dynamic bindings I think about Worker for Platform's Dynamic Dispatch binding, where it's just a singular binding you can use to get access to a worker/instance, ex:

    const worker = env.DISPATCHER.get("customer-worker-1");
    var responseFromUserScript = worker.fetch(req);

    const worker = env.DISPATCHER.get("customer-worker-1");
    var responseFromUserScript = worker.fetch(req);

Chaika•5/1/24, 9:27 PM

I don't think they've said much about dynamic bindings, but I imagine it'd be something like that where you've got maybe an entry binding (or maybe no entry binding at all and it's just part of the runtime/env object, all dynamic), but eitherway each binding wouldn't become part of your env or stick around, except for during that request

Frank•5/1/24, 9:32 PM

any idea how migrations work when u have so many dbs

RubiOP•5/2/24, 12:38 AM

basically a Go script to execute .sql concurrently on d1 rest api

Kkian `wrangler d1 export`

Softbear•5/2/24, 5:02 AM

Missed that news

CChaika when I think about dynamic bindings I think about Worker for Platform's Dynamic ...

Beny•5/2/24, 7:12 AM

I created something like this with Worker for Platforms. It uploads a new script with the binding that is essentially a D1 proxy over RPC. The only downside is that it can't be used immediately because the dispatcher doesn't think the script is valid straight away. The only solution I could get to work was implementing a delay of a few seconds before the script is usable. Apart from that it works very well

Sam•5/2/24, 8:12 PM

Hey all I wanted to echo https://community.cloudflare.com/t/different-errors-on-d1-randomly-happening-once-in-a-while/599421 -- Every few days now I get a "D1_ERROR: D1 storage operation exceeded timeout which caused object to be reset."

Is this a known issue? Very limited results on google & none in Discord

(

fdb7837214fdd80e018b0bf3ae6355c9

fdb7837214fdd80e018b0bf3ae6355c9

<redacted>-nx-production

<redacted>-nx-production

)

malone•5/2/24, 9:11 PM

I'm a bit confused with the upgrade process. When i ran wrangler d1 info, it said 'beta' instead of alpha and I didn't do anything yet.

malone•5/2/24, 9:11 PM

When i updated wrangler the version has been removed from the table entirely

malone•5/2/24, 9:11 PM

malone•5/2/24, 9:12 PM

with a brand new db i just created it also says beta

Unsmart•5/2/24, 9:13 PM

an alpha db would say alpha

Unsmart•5/2/24, 9:13 PM

beta is the most up to date version

malone•5/2/24, 9:13 PM

ok cool! so nothing to worry about. thanks @Unsmart | Tech debt

ALECKS•5/3/24, 12:26 AM

Is it possible to connect to a D1 database programmatically, from a remote host? For ex, I want to run a js script on my machine that connects to a D1 instance and does some stuff. I want to do this with JS, instead of .sql files, which I know I can run through wrangler d1 execute

Original message was deleted

forslunds•5/3/24, 3:05 AM

The local is a sqlite file inside wrangler folder. Open it in a sqlite program and export

stibbs•5/3/24, 3:35 AM

Apologies if this has been asked before, but is there a way to run down migrations? Or is it advised to:
1) create a new up migration to unwind a change [I am assuming this is the answer]; or
2) revert to prior state via time travel?

Frank•5/3/24, 4:59 AM

https://alexgarcia.xyz/blog/2024/building-new-vector-search-sqlite/index.html
pretty cool if this works well and eventually lands in D1 land

I'm writing a new vector search SQLite Extension

sqlite-vec is an new vector search SQLite extension, coming soon!

Kysan•5/3/24, 6:28 PM

https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database
what is the rate limit for the http api endpoint ?

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

Kysan•5/3/24, 6:29 PM

1200 requests per 5 minutes ty

KKysan 1200 requests per 5 minutes ty

Chaika•5/3/24, 6:32 PM

yea and that's shared among all endpoints/dashboard/etc for your user, some endpoints like purge by url have special ratelimits as well, but d1 doesn't

Chaika•5/3/24, 6:33 PM

?d1-remote-connections

Flare•5/3/24, 6:33 PM

Accessing D1 from outside workers:

D1 is based on SQLite, which does not natively support remote connections and isn't MySQL/PostgreSQL wire compatible, nor does D1 itself offer that functionality. As such, your options are:

1 -> The Cloudflare API Query Endpoint (https://developers.cloudflare.com/api/operations/cloudflare-d1-query-database), with the upside of needing less setup, all requests have to go back to Cloudflare's core, so request latency is far from optimal from some locations. You would also be subject to the API Rate limit of 1200 requests per 5 minutes.

2 -> Create a worker to proxy requests to your D1 Database. Either tightly bound to your application/schema or loose and just accepting any query. This would allow your worker to execute potentially closer to your D1 Database and avoid the api rate limit.

Miyagi•5/3/24, 8:18 PM

Hello, I have had success with the D1 in development all the way until production deployment. I have bound it using the updated syntax from docs (Pages). When i attempt to query when deployed, It crashes with Application Error.

Works just fine in wrangler preview.

AccountId: 77e1b1d919d86d253e4967517aab4437

Steved•5/3/24, 10:11 PM

Does anyone have an example of storing binary data in D1 using Drizzle?
Drizzle seems to be expecting a 'Buffer' type, but when I pass one in I'm getting an assert in Miniflare where it isn't expecting a Buffer in ArrayBufferView (ALLOWED_ARRAY_BUFFER_VIEW_CONSTRUCTORS). It's expecting one of the typed array names. At this point I can't tell whether the issue is in Drizzle or Miniflare (or devalue package).

Steved•5/3/24, 10:19 PM

If I ignore types and just pass a Uint8Array through it works.

its basically an sqlite file, maybe you need to do some load test

Similar Threads

Similar Threads

Similar Threads