Thank you! Then I think I'm beginning to understand the limitations and differences between DO SQLit
Thank you! Then I think I'm beginning to understand the limitations and differences between DO SQLite and D1 better 
ZVery exciting stuff!
LLocal vs remote is the key difference. Everything else derives from that.
You choose D1 for the usability layer (UI, insights for slow queries and metrics, wrangler commands, import/export), and you choose DO SQLite if you really want the local access + dynamic bindings benefits and are OK building the usability layer yourself (or using a library that provides it for you).
You choose D1 for the usability layer (UI, insights for slow queries and metrics, wrangler commands, import/export), and you choose DO SQLite if you really want the local access + dynamic bindings benefits and are OK building the usability layer yourself (or using a library that provides it for you).
ZNice! Thank you!
Here's where I found the timeline btw: https://blog.cloudflare.com/sqlite-in-durable-objects/#what-about-d1
Here's where I found the timeline btw: https://blog.cloudflare.com/sqlite-in-durable-objects/#what-about-d1
By the end of 2024, D1 will support automatic read replication for scalability and low-latency access around the world. If this managed model appeals to you, use D1.
JI use DO per user, but in my case it works nicely since user accounts are separate tenants. Each user registration runs the schema startup script / migrations to create the db on the fly. I ensure the DO ID is deterministic based on some user specific key. Every subsequent login / request, the key is unpacked / looked up and then I can re-access the DO dynamically.
PHello, I am trying to run a query in D1 with multiple
Is there any reason for setting this to so low(like any other limitations) and is there way to increase this now or in the future with D1 or SQLite in DO?
UNION ALL SELECT terms. According to SQLite documentation, the number of terms in the query is limited by the value SQLITE_MAX_COMPOUND_SELECT which defaults to 500. But in D1 this is set to 5 and any query with more than 5 terms (like for inserting multiple values at once) fail with too many terms in compound SELECT. Is there any reason for setting this to so low(like any other limitations) and is there way to increase this now or in the future with D1 or SQLite in DO?
PI see thanks. Also saw its a recursive algorithm so makes sense now
Aive been seeing a bunch of
D1_ERROR: Network connection lost. errors being thrown since friday. Any idea if this is on d1 or my side?Aa similar issue appears to be happening with durable objects itself. Isn't D1 based on DO? so might actually be a DO issue?
KSorry for the confusion.
The production has 21 records, see the attached. And the local in my machine has 5 records. If I am running the local using the wrangler dev + cloudflare since I am using opennext it behaves as expected, in the UI it only shows 5 records. but when i deploy with the wrangler deploy + cloudflare, say I fetched 21 records from an API. It still displays the 5 records from the local.
I used the --remote flag when migrating but still displaying the 5 records from the local.
I am using swr to reflect the real time changes from the database and storage. Not sure if this affects it.
The production has 21 records, see the attached. And the local in my machine has 5 records. If I am running the local using the wrangler dev + cloudflare since I am using opennext it behaves as expected, in the UI it only shows 5 records. but when i deploy with the wrangler deploy + cloudflare, say I fetched 21 records from an API. It still displays the 5 records from the local.
I used the --remote flag when migrating but still displaying the 5 records from the local.
I am using swr to reflect the real time changes from the database and storage. Not sure if this affects it.
LIf the database in the cloudflare dashboard shows you the right rows, then it means your D1 database is correct.
So, the issue is in the code build process.
I don't use opennext or swr so cannot comment for those, but make sure when you build your application before deploying that it does not use the local database to build up the content of your site in some way and you fetch the data at runtime using the correct database in production. If you try to fetch at build time, then it's very likely using your local database. Again, just a guess though, so check your build process.
So, the issue is in the code build process.
I don't use opennext or swr so cannot comment for those, but make sure when you build your application before deploying that it does not use the local database to build up the content of your site in some way and you fetch the data at runtime using the correct database in production. If you try to fetch at build time, then it's very likely using your local database. Again, just a guess though, so check your build process.
KI think it is. Cos when I delete the local generated sql from .wrangler. it displays the 21 records in the UI.
Kthis is what solve the issue
import { defineConfig } from "drizzle-kit";
import type { Config } from "drizzle-kit";
import fs from "node:fs";
import path from "node:path";
function getLocalD1DB() {
try {
const basePath = path.resolve(".wrangler");
const dbFile = fs
.readdirSync(basePath, { encoding: "utf-8", recursive: true })
.find((f) => f.endsWith(".sqlite"));
if (!dbFile) {
throw new Error(
}
const url = path.resolve(basePath, dbFile);
return url;
} catch (err) {
console.log(
}
}
export default defineConfig({
dialect: "sqlite",
schema: "./schemas/drizzle.ts",
out: "./drizzle",
...(process.env.CLOUDFLARE_D1_ACCOUNT_ID
? {
driver: "d1-http",
dbCredentials: {
accountId: process.env.CLOUDFLARE_D1_ACCOUNT_ID,
databaseId: process.env.DATABASE,
token: process.env.CLOUDFLARE_D1_API_TOKEN,
},
}
: {
dbCredentials: {
url: getLocalD1DB(),
},
}),
}) satisfies Config;
import { defineConfig } from "drizzle-kit";
import type { Config } from "drizzle-kit";
import fs from "node:fs";
import path from "node:path";
function getLocalD1DB() {
try {
const basePath = path.resolve(".wrangler");
const dbFile = fs
.readdirSync(basePath, { encoding: "utf-8", recursive: true })
.find((f) => f.endsWith(".sqlite"));
if (!dbFile) {
throw new Error(
.sqlite file not found in ${basePath});}
const url = path.resolve(basePath, dbFile);
return url;
} catch (err) {
console.log(
Error ${err});}
}
export default defineConfig({
dialect: "sqlite",
schema: "./schemas/drizzle.ts",
out: "./drizzle",
...(process.env.CLOUDFLARE_D1_ACCOUNT_ID
? {
driver: "d1-http",
dbCredentials: {
accountId: process.env.CLOUDFLARE_D1_ACCOUNT_ID,
databaseId: process.env.DATABASE,
token: process.env.CLOUDFLARE_D1_API_TOKEN,
},
}
: {
dbCredentials: {
url: getLocalD1DB(),
},
}),
}) satisfies Config;
Athe error has changed to
D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.
CAnything interesting with metrics? Does it happen even on simple queries, consistently, etc?
The Network Connection Lost error just does happen every so often..unfortunately, but it depends on frequency and such.
The Network Connection Lost error just does happen every so often..unfortunately, but it depends on frequency and such.
Aits not consistent. there doesn't appear to be anything out of the ordinary on the db's metrics
Awell, that error specifically seems to only happen on one query, but when it happens on that one query seems random
ACindex on timestamp? (or is table small enough to not need it?)
npx wrangler d1 insights <db> can be help as well to look through expensive queries
npx wrangler d1 insights <db> can be help as well to look through expensive queries
Athat table should get cleared at the end of every week, so I would be suprised if it had more than a few thousand rows at the absolute most
Ayeah that table currently has 0 rows
TI'm working on some tooling which is handling very sensitive data on behalf of my customers, along with user-defined workers (Workers for Platforms). But I'd like to ensure that I never have access to that data. Obviously I can encrypt the data with a key that I dont have, but I need workers to be able to access the data, and I dont want the user to have to manage that key either. Any experience / advice for doing something like this?
Mwhats your database_id?
LIf you want encryption then someone has to have the key, otherwise you won't be able to decrypt the data. You could in theory add a secret in your worker and "lose it" so that it's only in the worker dashboard, but then if you ever need to put that into a different worker, or decrypt the data outside of that worker you cannot do anything, so do NOT do this.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.
A03bb813c-a64f-4a34-9483-dc285ab59ba2
Athe error probably wont happen again until tonight at the earliest. Will most likely happen tomorrow around December 21, 2024, as that is my peak time and it seems to happen more often with more requests
TYeah I had been thinking about trying something like that. I feel like there are some interesting capabilities that could be opened up with some tooling like this that wouldn't be too hard to implement. Basically having the worker fully own the key and access to it. Would also need to think through how the worker code gets upgraded...
Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)
Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)
Twhat you mentioned about "losing" the key is kind of how it works with Lit. I provision a new keypair, initially owned by me - so I can update the access rules/code. I dont actually get the key itself (it is never exposed), but I have full access to use it and update access rules. Then once it's created and set up properly I transfer the ownership to itself.
You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.
You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.
JBeen messing around with D1 for storing a websites searchable product list. However, any SQL query reads all rows and at 19k rows that would eat away at daily limits.
Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?
Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?
Owhat is the write concurrency for d1? i have an event store and a batching queue system with deduplication via d1, but i'm not sure if i should be writing to a queue which writes to d1 after or if it should instead be writing to d1 itself. at peak i can get say 1k updates/s - therefore not sure if batching via queues is a better idea here
MIf your FTS queries scan multiple columns you need to ensure your indexes cover those columns: https://developers.cloudflare.com/d1/best-practices/use-indexes/
Cloudflare Docs
Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) the database has to scan when running a query.
LIf this 1k writes/s is for a single database, and you could be writing many rows each time, you are at the border of what you can do. It means you should not spend more than 1ms per query. Doable if they are selects, but for writes it will really depend on many factors.
Batching even more earlier, or using a few database instances would give you a lot more room and flexibility.
Batching even more earlier, or using a few database instances would give you a lot more room and flexibility.
Oalright makes sense, thanks 
TIs there anything you can do to mitigate ? The web interface is also unresponsive for this database.
id:
id:
Dhappens fairly regularly, all you can do is re-attempt your query
JYeah, hopefully we'll get some better documentation around error rates and safely retriable ones as per https://github.com/cloudflare/cloudflare-docs/issues/18485. Until then, unfortunately all you can do is retry or use another more stable database provider if the error rates aren't acceptable for your use-case.
TI tried that obviously, but my database seems to be locked up in this state. It’s all it gives back for the past 14 hours. Can’t connect to it through web, wrangler or code.
Dah, sounds like it's very broken, my D1 databases do that periodically but usually only briefly
KWhen creating a D1 database, I can provide a location hint like EEUR. However, this region is big, and I want to be close to my users. Is there any way of figuring out which data center actually hosts the database, so I can retry until I actually hit the right location? I know about the
d1 info option, but this only returns EEUR.KMy understanding is that D1 is not currently replicated?
KRight, but presumably there is a "primary" data center for the database.
KFor one database we have in the WEUR region, it seems to always return MRS in the smart placement header, even though there are several much closer data centers.
KI find it utterly baffling that this information is obscured from us.
LCan you check your database now? It should be fixed.
TIt works indeed! Thank you 
Tthis happens because the API for the sqlite d1 file in miniflare is slightly different from the managed postgres api in live. Please share your solution because mine is really dumb
