youd have to chain them via queue or worker request likely
youd have to chain them via queue or worker request likely
Jthats just one way, you could limit your delete another way, just add additional bounds to the delete where clause and step through it
Dcheers, will step through it with some pagiantion client / server interplay
JJid bet you could bypass the 100 id limit with a subquery, could just be an api limitation, not sqlite
LThe limit is for how many variables you can
bind()
DYou may like using Workers + Hyperdrive 
Dbut copying 2 million rows from one table to another i havent gotten to work yet, this presumably then hits some query timeout limits?
RAre there any tips to decrease the rate of
I am see a concerning increase in these that is causing issues in my application.
D1_ERROR: Network connection lostI am see a concerning increase in these that is causing issues in my application.
Dnot that i'm aware of. i get about 1-2 per day that i monitor on my worker. Only real solution is re-attempt the query
AI‘m looking for an architecture example of how to scale D1 per user as listed within the First Couple of lines in the documentation. Can someone help me?
LThere isn't much advertised architecture really. The idea is that you create more than 1 database depending on your needs.
If your data can be sharded in a manageable way, you can create N databases, and shard your data onto those databases.
The extreme case is when you have a new database for each individual user/account, but right now you need a bit of extra work to create and attach those databases dynamically through the REST API.
My suggestion is the first one, a couple of databases (5-20) and sharding your data across them depending on your load/access needs. But as always, it depends what you want to do, and what you data looks like, and how your access patterns look like.
If your data can be sharded in a manageable way, you can create N databases, and shard your data onto those databases.
The extreme case is when you have a new database for each individual user/account, but right now you need a bit of extra work to create and attach those databases dynamically through the REST API.
My suggestion is the first one, a couple of databases (5-20) and sharding your data across them depending on your load/access needs. But as always, it depends what you want to do, and what you data looks like, and how your access patterns look like.
ZHas sessions for D1 shipped? Would the session api allow for transactions since it's sequential consistent for that session?
LBy sessions, do you refer to the read replication feature? Which blog post do you have in mind?
ZYes, the read replication feature.
This blog post:
https://blog.cloudflare.com/building-d1-a-global-database/
But it actually doesn't mention a timeline there, I don't know where I got end of the year from. I've been reading up on DO SQLite and D1 quite a lot today, so it might just be wishful thinking from my side
This blog post:
https://blog.cloudflare.com/building-d1-a-global-database/
But it actually doesn't mention a timeline there, I don't know where I got end of the year from. I've been reading up on DO SQLite and D1 quite a lot today, so it might just be wishful thinking from my side
LYes, this sessions API and read replication is coming soon-ish. And transactions/writes will continue working fine too.
ZTransactions in D1 today requires the use of .batch(), right?
I was hoping that the transaction api from DO SQLite would be available within sessions as well (https://github.com/outerbase/starbasedb/issues/12).
This is not something that has been mentioned anywhere (other than this blog post from 2022: https://blog.cloudflare.com/whats-new-with-d1/).
I was hoping that the transaction api from DO SQLite would be available within sessions as well (https://github.com/outerbase/starbasedb/issues/12).
This is not something that has been mentioned anywhere (other than this blog post from 2022: https://blog.cloudflare.com/whats-new-with-d1/).
LThe DO transactions API works really well because you access a local SQLite database, so you get very good performance even with tens/hundreds of queries. Unless you mess it up with async IO yourself.
D1 is a remote database, away from your Worker. This transaction API doesn't work well with that since you would be making multiple remote calls each time. Not saying there isn't an API that can work OK-ish, but that one isn't it.
D1 is a remote database, away from your Worker. This transaction API doesn't work well with that since you would be making multiple remote calls each time. Not saying there isn't an API that can work OK-ish, but that one isn't it.
ZThank you! Then I think I'm beginning to understand the limitations and differences between DO SQLite and D1 better
ZVery exciting stuff!
LLocal vs remote is the key difference. Everything else derives from that.
You choose D1 for the usability layer (UI, insights for slow queries and metrics, wrangler commands, import/export), and you choose DO SQLite if you really want the local access + dynamic bindings benefits and are OK building the usability layer yourself (or using a library that provides it for you).
You choose D1 for the usability layer (UI, insights for slow queries and metrics, wrangler commands, import/export), and you choose DO SQLite if you really want the local access + dynamic bindings benefits and are OK building the usability layer yourself (or using a library that provides it for you).
ZNice! Thank you!
Here's where I found the timeline btw: https://blog.cloudflare.com/sqlite-in-durable-objects/#what-about-d1
Here's where I found the timeline btw: https://blog.cloudflare.com/sqlite-in-durable-objects/#what-about-d1
By the end of 2024, D1 will support automatic read replication for scalability and low-latency access around the world. If this managed model appeals to you, use D1.
JI use DO per user, but in my case it works nicely since user accounts are separate tenants. Each user registration runs the schema startup script / migrations to create the db on the fly. I ensure the DO ID is deterministic based on some user specific key. Every subsequent login / request, the key is unpacked / looked up and then I can re-access the DO dynamically.
PHello, I am trying to run a query in D1 with multiple
Is there any reason for setting this to so low(like any other limitations) and is there way to increase this now or in the future with D1 or SQLite in DO?
UNION ALL SELECTSQLITE_MAX_COMPOUND_SELECTtoo many terms in compound SELECTIs there any reason for setting this to so low(like any other limitations) and is there way to increase this now or in the future with D1 or SQLite in DO?
PI see thanks. Also saw its a recursive algorithm so makes sense now
Aive been seeing a bunch of
D1_ERROR: Network connection lost.a similar issue appears to be happening with durable objects itself. Isn't D1 based on DO? so might actually be a DO issue?
KSorry for the confusion.
The production has 21 records, see the attached. And the local in my machine has 5 records. If I am running the local using the wrangler dev + cloudflare since I am using opennext it behaves as expected, in the UI it only shows 5 records. but when i deploy with the wrangler deploy + cloudflare, say I fetched 21 records from an API. It still displays the 5 records from the local.
I used the --remote flag when migrating but still displaying the 5 records from the local.
I am using swr to reflect the real time changes from the database and storage. Not sure if this affects it.
The production has 21 records, see the attached. And the local in my machine has 5 records. If I am running the local using the wrangler dev + cloudflare since I am using opennext it behaves as expected, in the UI it only shows 5 records. but when i deploy with the wrangler deploy + cloudflare, say I fetched 21 records from an API. It still displays the 5 records from the local.
I used the --remote flag when migrating but still displaying the 5 records from the local.
I am using swr to reflect the real time changes from the database and storage. Not sure if this affects it.
LIf the database in the cloudflare dashboard shows you the right rows, then it means your D1 database is correct.
So, the issue is in the code build process.
I don't use opennext or swr so cannot comment for those, but make sure when you build your application before deploying that it does not use the local database to build up the content of your site in some way and you fetch the data at runtime using the correct database in production. If you try to fetch at build time, then it's very likely using your local database. Again, just a guess though, so check your build process.
So, the issue is in the code build process.
I don't use opennext or swr so cannot comment for those, but make sure when you build your application before deploying that it does not use the local database to build up the content of your site in some way and you fetch the data at runtime using the correct database in production. If you try to fetch at build time, then it's very likely using your local database. Again, just a guess though, so check your build process.
KI think it is. Cos when I delete the local generated sql from .wrangler. it displays the 21 records in the UI.
Kthis is what solve the issue
import { defineConfig } from "drizzle-kit";
import type { Config } from "drizzle-kit";
import fs from "node:fs";
import path from "node:path";
function getLocalD1DB() {
try {
const basePath = path.resolve(".wrangler");
const dbFile = fs
.readdirSync(basePath, { encoding: "utf-8", recursive: true })
.find((f) => f.endsWith(".sqlite"));
if (!dbFile) {
throw new Error(
}
const url = path.resolve(basePath, dbFile);
return url;
} catch (err) {
console.log(
}
}
export default defineConfig({
dialect: "sqlite",
schema: "./schemas/drizzle.ts",
out: "./drizzle",
...(process.env.CLOUDFLARE_D1_ACCOUNT_ID
? {
driver: "d1-http",
dbCredentials: {
accountId: process.env.CLOUDFLARE_D1_ACCOUNT_ID,
databaseId: process.env.DATABASE,
token: process.env.CLOUDFLARE_D1_API_TOKEN,
},
}
: {
dbCredentials: {
url: getLocalD1DB(),
},
}),
}) satisfies Config;
import { defineConfig } from "drizzle-kit";
import type { Config } from "drizzle-kit";
import fs from "node:fs";
import path from "node:path";
function getLocalD1DB() {
try {
const basePath = path.resolve(".wrangler");
const dbFile = fs
.readdirSync(basePath, { encoding: "utf-8", recursive: true })
.find((f) => f.endsWith(".sqlite"));
if (!dbFile) {
throw new Error(
.sqlite file not found in ${basePath}}
const url = path.resolve(basePath, dbFile);
return url;
} catch (err) {
console.log(
Error ${err}}
}
export default defineConfig({
dialect: "sqlite",
schema: "./schemas/drizzle.ts",
out: "./drizzle",
...(process.env.CLOUDFLARE_D1_ACCOUNT_ID
? {
driver: "d1-http",
dbCredentials: {
accountId: process.env.CLOUDFLARE_D1_ACCOUNT_ID,
databaseId: process.env.DATABASE,
token: process.env.CLOUDFLARE_D1_API_TOKEN,
},
}
: {
dbCredentials: {
url: getLocalD1DB(),
},
}),
}) satisfies Config;
Athe error has changed to
D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.
CAnything interesting with metrics? Does it happen even on simple queries, consistently, etc?
The Network Connection Lost error just does happen every so often..unfortunately, but it depends on frequency and such.
The Network Connection Lost error just does happen every so often..unfortunately, but it depends on frequency and such.
Aits not consistent. there doesn't appear to be anything out of the ordinary on the db's metrics
Awell, that error specifically seems to only happen on one query, but when it happens on that one query seems random
ACindex on timestamp? (or is table small enough to not need it?)
npx wrangler d1 insights <db> can be help as well to look through expensive queries
npx wrangler d1 insights <db> can be help as well to look through expensive queries
Athat table should get cleared at the end of every week, so I would be suprised if it had more than a few thousand rows at the absolute most
Ayeah that table currently has 0 rows
TI'm working on some tooling which is handling very sensitive data on behalf of my customers, along with user-defined workers (Workers for Platforms). But I'd like to ensure that I never have access to that data. Obviously I can encrypt the data with a key that I dont have, but I need workers to be able to access the data, and I dont want the user to have to manage that key either. Any experience / advice for doing something like this?
Mwhats your database_id?
LIf you want encryption then someone has to have the key, otherwise you won't be able to decrypt the data. You could in theory add a secret in your worker and "lose it" so that it's only in the worker dashboard, but then if you ever need to put that into a different worker, or decrypt the data outside of that worker you cannot do anything, so do NOT do this.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.
A03bb813c-a64f-4a34-9483-dc285ab59ba2
Athe error probably wont happen again until tonight at the earliest. Will most likely happen tomorrow around December 21, 2024, as that is my peak time and it seems to happen more often with more requests
TYeah I had been thinking about trying something like that. I feel like there are some interesting capabilities that could be opened up with some tooling like this that wouldn't be too hard to implement. Basically having the worker fully own the key and access to it. Would also need to think through how the worker code gets upgraded...
Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)
Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)
Twhat you mentioned about "losing" the key is kind of how it works with Lit. I provision a new keypair, initially owned by me - so I can update the access rules/code. I dont actually get the key itself (it is never exposed), but I have full access to use it and update access rules. Then once it's created and set up properly I transfer the ownership to itself.
You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.
You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.
JBeen messing around with D1 for storing a websites searchable product list. However, any SQL query reads all rows and at 19k rows that would eat away at daily limits.
Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?
Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?
Owhat is the write concurrency for d1? i have an event store and a batching queue system with deduplication via d1, but i'm not sure if i should be writing to a queue which writes to d1 after or if it should instead be writing to d1 itself. at peak i can get say 1k updates/s - therefore not sure if batching via queues is a better idea here
MIf your FTS queries scan multiple columns you need to ensure your indexes cover those columns: https://developers.cloudflare.com/d1/best-practices/use-indexes/
Cloudflare Docs
Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) the database has to scan when running a query.
