I think it is. Cos when I delete the local generated sql from .wrangler. it displays the 21 records
I think it is. Cos when I delete the local generated sql from .wrangler. it displays the 21 records in the UI.
Kthis is what solve the issue
import { defineConfig } from "drizzle-kit";
import type { Config } from "drizzle-kit";
import fs from "node:fs";
import path from "node:path";
function getLocalD1DB() {
try {
const basePath = path.resolve(".wrangler");
const dbFile = fs
.readdirSync(basePath, { encoding: "utf-8", recursive: true })
.find((f) => f.endsWith(".sqlite"));
if (!dbFile) {
throw new Error(
}
const url = path.resolve(basePath, dbFile);
return url;
} catch (err) {
console.log(
}
}
export default defineConfig({
dialect: "sqlite",
schema: "./schemas/drizzle.ts",
out: "./drizzle",
...(process.env.CLOUDFLARE_D1_ACCOUNT_ID
? {
driver: "d1-http",
dbCredentials: {
accountId: process.env.CLOUDFLARE_D1_ACCOUNT_ID,
databaseId: process.env.DATABASE,
token: process.env.CLOUDFLARE_D1_API_TOKEN,
},
}
: {
dbCredentials: {
url: getLocalD1DB(),
},
}),
}) satisfies Config;
import { defineConfig } from "drizzle-kit";
import type { Config } from "drizzle-kit";
import fs from "node:fs";
import path from "node:path";
function getLocalD1DB() {
try {
const basePath = path.resolve(".wrangler");
const dbFile = fs
.readdirSync(basePath, { encoding: "utf-8", recursive: true })
.find((f) => f.endsWith(".sqlite"));
if (!dbFile) {
throw new Error(
.sqlite file not found in ${basePath});}
const url = path.resolve(basePath, dbFile);
return url;
} catch (err) {
console.log(
Error ${err});}
}
export default defineConfig({
dialect: "sqlite",
schema: "./schemas/drizzle.ts",
out: "./drizzle",
...(process.env.CLOUDFLARE_D1_ACCOUNT_ID
? {
driver: "d1-http",
dbCredentials: {
accountId: process.env.CLOUDFLARE_D1_ACCOUNT_ID,
databaseId: process.env.DATABASE,
token: process.env.CLOUDFLARE_D1_API_TOKEN,
},
}
: {
dbCredentials: {
url: getLocalD1DB(),
},
}),
}) satisfies Config;
Athe error has changed to
D1_ERROR: D1 DB storage operation exceeded timeout which caused object to be reset.
CAnything interesting with metrics? Does it happen even on simple queries, consistently, etc?
The Network Connection Lost error just does happen every so often..unfortunately, but it depends on frequency and such.
The Network Connection Lost error just does happen every so often..unfortunately, but it depends on frequency and such.
Aits not consistent. there doesn't appear to be anything out of the ordinary on the db's metrics
Awell, that error specifically seems to only happen on one query, but when it happens on that one query seems random
ACindex on timestamp? (or is table small enough to not need it?)
npx wrangler d1 insights <db> can be help as well to look through expensive queries
npx wrangler d1 insights <db> can be help as well to look through expensive queries
Athat table should get cleared at the end of every week, so I would be suprised if it had more than a few thousand rows at the absolute most
Ayeah that table currently has 0 rows
TI'm working on some tooling which is handling very sensitive data on behalf of my customers, along with user-defined workers (Workers for Platforms). But I'd like to ensure that I never have access to that data. Obviously I can encrypt the data with a key that I dont have, but I need workers to be able to access the data, and I dont want the user to have to manage that key either. Any experience / advice for doing something like this?
Mwhats your database_id?
LIf you want encryption then someone has to have the key, otherwise you won't be able to decrypt the data. You could in theory add a secret in your worker and "lose it" so that it's only in the worker dashboard, but then if you ever need to put that into a different worker, or decrypt the data outside of that worker you cannot do anything, so do NOT do this.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.
A03bb813c-a64f-4a34-9483-dc285ab59ba2
Athe error probably wont happen again until tonight at the earliest. Will most likely happen tomorrow around December 21, 2024, as that is my peak time and it seems to happen more often with more requests
TYeah I had been thinking about trying something like that. I feel like there are some interesting capabilities that could be opened up with some tooling like this that wouldn't be too hard to implement. Basically having the worker fully own the key and access to it. Would also need to think through how the worker code gets upgraded...
Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)
Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)
Twhat you mentioned about "losing" the key is kind of how it works with Lit. I provision a new keypair, initially owned by me - so I can update the access rules/code. I dont actually get the key itself (it is never exposed), but I have full access to use it and update access rules. Then once it's created and set up properly I transfer the ownership to itself.
You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.
You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.
JBeen messing around with D1 for storing a websites searchable product list. However, any SQL query reads all rows and at 19k rows that would eat away at daily limits.
Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?
Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?
Owhat is the write concurrency for d1? i have an event store and a batching queue system with deduplication via d1, but i'm not sure if i should be writing to a queue which writes to d1 after or if it should instead be writing to d1 itself. at peak i can get say 1k updates/s - therefore not sure if batching via queues is a better idea here
MIf your FTS queries scan multiple columns you need to ensure your indexes cover those columns: https://developers.cloudflare.com/d1/best-practices/use-indexes/
Cloudflare Docs
Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) the database has to scan when running a query.
LIf this 1k writes/s is for a single database, and you could be writing many rows each time, you are at the border of what you can do. It means you should not spend more than 1ms per query. Doable if they are selects, but for writes it will really depend on many factors.
Batching even more earlier, or using a few database instances would give you a lot more room and flexibility.
Batching even more earlier, or using a few database instances would give you a lot more room and flexibility.
Oalright makes sense, thanks 
TIs there anything you can do to mitigate ? The web interface is also unresponsive for this database.
id:
id:
Dhappens fairly regularly, all you can do is re-attempt your query
JYeah, hopefully we'll get some better documentation around error rates and safely retriable ones as per https://github.com/cloudflare/cloudflare-docs/issues/18485. Until then, unfortunately all you can do is retry or use another more stable database provider if the error rates aren't acceptable for your use-case.
TI tried that obviously, but my database seems to be locked up in this state. It’s all it gives back for the past 14 hours. Can’t connect to it through web, wrangler or code.
Dah, sounds like it's very broken, my D1 databases do that periodically but usually only briefly
KWhen creating a D1 database, I can provide a location hint like EEUR. However, this region is big, and I want to be close to my users. Is there any way of figuring out which data center actually hosts the database, so I can retry until I actually hit the right location? I know about the
d1 info option, but this only returns EEUR.KMy understanding is that D1 is not currently replicated?
KRight, but presumably there is a "primary" data center for the database.
KFor one database we have in the WEUR region, it seems to always return MRS in the smart placement header, even though there are several much closer data centers.
KI find it utterly baffling that this information is obscured from us.
LCan you check your database now? It should be fixed.
TIt works indeed! Thank you 
Tthis happens because the API for the sqlite d1 file in miniflare is slightly different from the managed postgres api in live. Please share your solution because mine is really dumb
MIf D1 is distributed (read only) why so much diference from just a SELECT from the EU and the US?
HIt’s not, D1 is single-location for now
MIt's there a timeline? I am migrating everything because the documentation says it's distributed
MI thought you could have many instances in each region or something like that
MThere is a whole section in D1 documentation about read replicas. Is this not true??
MI mean.. they market as D1 is distributed and read replicas.. I expected you know, they have read replicas.
HIt’s a planned feature, but it doesn’t exist yet
MExactly the same query from different locations.
MCan we know if it's planned for Q1 or more into the future?
