I'm working on some tooling which is handling very sensitive data on behalf of my customers, along w

I'm working on some tooling which is handling very sensitive data on behalf of my customers, along with user-defined workers (Workers for Platforms). But I'd like to ensure that I never have access to that data. Obviously I can encrypt the data with a key that I dont have, but I need workers to be able to access the data, and I dont want the user to have to manage that key either. Any experience / advice for doing something like this?

Aajgeiss0702 the error has changed to `D1_ERROR: D1 DB storage operation exceeded timeout whi...

Max (@rozenmd)•12/19/24, 9:49 AM

whats your database_id?

Ttheo I'm working on some tooling which is handling very sensitive data on behalf of m...

lambrospetrou•12/19/24, 10:02 AM

If you want encryption then someone has to have the key, otherwise you won't be able to decrypt the data. You could in theory add a secret in your worker and "lose it" so that it's only in the worker dashboard, but then if you ever need to put that into a different worker, or decrypt the data outside of that worker you cannot do anything, so do NOT do this.
You could infer keys from something unique to each user that they anyway provide into your application, as an alternative.

MMax (@rozenmd)whats your database_id?

ajgeiss0702•12/19/24, 4:26 PM

03bb813c-a64f-4a34-9483-dc285ab59ba2

ajgeiss0702•12/19/24, 4:28 PM

the error probably wont happen again until tonight at the earliest. Will most likely happen tomorrow around December 21, 2024, as that is my peak time and it seems to happen more often with more requests

Llambrospetrou If you want encryption then someone has to have the key, otherwise you won't be ...

theoOP•12/19/24, 6:46 PM

Yeah I had been thinking about trying something like that. I feel like there are some interesting capabilities that could be opened up with some tooling like this that wouldn't be too hard to implement. Basically having the worker fully own the key and access to it. Would also need to think through how the worker code gets upgraded...

Basically trying to get some of the benefits of using a decentralized programmable key pair without having to involve blockchain... Relying on the Cloudflare's builtin security rather than something much more complicated. (see https://www.litprotocol.com/)

theoOP•12/19/24, 6:51 PM

what you mentioned about "losing" the key is kind of how it works with Lit. I provision a new keypair, initially owned by me - so I can update the access rules/code. I dont actually get the key itself (it is never exposed), but I have full access to use it and update access rules. Then once it's created and set up properly I transfer the ownership to itself.

You could imagine something similar with a worker. It could implement some kind of standardized API, and then be put into a mode like this. In order to update the worker - youd need to get some kind of token that only the worker can returns - so it can manage access of its own updates.

Jaykubz•12/19/24, 8:32 PM

Been messing around with D1 for storing a websites searchable product list. However, any SQL query reads all rows and at 19k rows that would eat away at daily limits.

Was messing with FTS for a table and my worker api calls say it is only querying the exact amount of rows. On the back end in one of the FTS tables are there 19k row reads happening and only ~40 on the main table due to indexs?

oof2win2•12/20/24, 10:03 AM

what is the write concurrency for d1? i have an event store and a batching queue system with deduplication via d1, but i'm not sure if i should be writing to a queue which writes to d1 after or if it should instead be writing to d1 itself. at peak i can get say 1k updates/s - therefore not sure if batching via queues is a better idea here

JJaykubz Been messing around with D1 for storing a websites searchable product list. Howe...

Matt Silverlock•12/20/24, 1:06 PM

If your FTS queries scan multiple columns you need to ensure your indexes cover those columns: https://developers.cloudflare.com/d1/best-practices/use-indexes/

Cloudflare Docs

Use indexes · Cloudflare D1 docs

Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) the database has to scan when running a query.

JJaykubz Been messing around with D1 for storing a websites searchable product list. Howe...

Matt Silverlock•12/20/24, 1:09 PM

Also: https://www.sqlite.org/fts5.html#prefix_indexes

Ooof2win2 what is the write concurrency for d1? i have an event store and a batching queue...

lambrospetrou•12/20/24, 6:03 PM

If this 1k writes/s is for a single database, and you could be writing many rows each time, you are at the border of what you can do. It means you should not spend more than 1ms per query. Doable if they are selects, but for writes it will really depend on many factors.
Batching even more earlier, or using a few database instances would give you a lot more room and flexibility.

oof2win2•12/20/24, 6:09 PM

alright makes sense, thanks

TrulyUnlucky•12/20/24, 6:53 PM

Is there anything you can do to mitigate

D1_ERROR: Internal error while starting up D1 DB storage caused object to be reset.

D1_ERROR: Internal error while starting up D1 DB storage caused object to be reset.

? The web interface is also unresponsive for this database.

id:

6559cf5b-ea94-4b57-8f05-2228dc0075ea

6559cf5b-ea94-4b57-8f05-2228dc0075ea

TTrulyUnlucky Is there anything you can do to mitigate ``` D1_ERROR: Internal error while star...

Dean•12/20/24, 11:54 PM

happens fairly regularly, all you can do is re-attempt your query

James•12/20/24, 11:58 PM

Yeah, hopefully we'll get some better documentation around error rates and safely retriable ones as per https://github.com/cloudflare/cloudflare-docs/issues/18485. Until then, unfortunately all you can do is retry or use another more stable database provider if the error rates aren't acceptable for your use-case.

DDean happens fairly regularly, all you can do is re-attempt your query

TrulyUnlucky•12/21/24, 8:35 AM

I tried that obviously, but my database seems to be locked up in this state. It’s all it gives back for the past 14 hours. Can’t connect to it through web, wrangler or code.

TTrulyUnlucky I tried that obviously, but my database seems to be locked up in this state. It’...

Dean•12/21/24, 8:37 AM

ah, sounds like it's very broken, my D1 databases do that periodically but usually only briefly

Kyrre•12/21/24, 8:42 AM

When creating a D1 database, I can provide a location hint like EEUR. However, this region is big, and I want to be close to my users. Is there any way of figuring out which data center actually hosts the database, so I can retry until I actually hit the right location? I know about the

d1 info

d1 info

option, but this only returns EEUR.

Original message was deleted

Kyrre•12/21/24, 8:45 AM

My understanding is that D1 is not currently replicated?

Kyrre•12/21/24, 8:47 AM

Right, but presumably there is a "primary" data center for the database.

Kyrre•12/21/24, 8:49 AM

For one database we have in the WEUR region, it seems to always return MRS in the smart placement header, even though there are several much closer data centers.

Kyrre•12/21/24, 8:50 AM

I find it utterly baffling that this information is obscured from us.

TTrulyUnlucky Is there anything you can do to mitigate ``` D1_ERROR: Internal error while star...

lambrospetrou•12/21/24, 8:58 AM

Can you check your database now? It should be fixed.

Llambrospetrou Can you check your database now? It should be fixed.

TrulyUnlucky•12/21/24, 9:00 AM

It works indeed! Thank you

Bbutter hey guys i've been trying to setup a project that uses nextjs, hono, drizzle and...

Trojan•12/22/24, 1:17 PM

this happens because the API for the sqlite d1 file in miniflare is slightly different from the managed postgres api in live. Please share your solution because mine is really dumb

import { defineConfig, type Config } from "drizzle-kit";
import { parse as parseJsonc } from "comment-json";
import { existsSync, readFileSync } from "fs";
import { join } from "path";

const wranglerConfig = parseJsonc(
  readFileSync(join(__dirname, "wrangler.jsonc"), "utf-8"),
) as unknown as WranglerConfig;

const d1Config = wranglerConfig.d1_databases[0];

const env = Object.fromEntries(
  ((filePath) =>
    existsSync(filePath)
      ? readFileSync(filePath, "utf-8")
          .split("\n")
          .filter((line) => line.trim() && !line.startsWith("#"))
          .map((line) =>
            line
              .split(/=(.+)/)
              .filter(Boolean)
              .map((part) => part.trim()),
          )
          .filter((parts) => parts.length === 2)
          .map(([key, value]) => [key, value?.replace(/^["']|["']$/g, "")])
      : [])(join(process.cwd(), ".dev.vars")),
) as Record<string, string>;

/*
 * NOTE: Workaround to make drizzle studio work with D1.
 * https://kevinkipp.com/blog/going-full-stack-on-astro-with-cloudflare-d1-and-drizzle/
 * Github discussion: https://github.com/drizzle-team/drizzle-orm/discussions/1545#discussioncomment-8115423
 */
export default process.env["NODE_ENV"] === "production"
  ? defineConfig({
      dialect: "sqlite",
      driver: "d1-http",
      out: "drizzle",
      schema: "src/db/schema.ts",
      dbCredentials: {
        accountId: env["CF_ACCOUNT_ID"]!,
        token: env["CF_D1_TOKEN"]!,
        databaseId: d1Config!.database_id!,
      },
    } satisfies Config)
  : defineConfig({
      dialect: "sqlite",
      out: "drizzle",
      schema: "src/db/schema.ts",
      dbCredentials: {
        url: env["DB_LOCAL_PATH"]!,
      },
    } satisfies Config);

import { defineConfig, type Config } from "drizzle-kit";
import { parse as parseJsonc } from "comment-json";
import { existsSync, readFileSync } from "fs";
import { join } from "path";

const wranglerConfig = parseJsonc(
  readFileSync(join(__dirname, "wrangler.jsonc"), "utf-8"),
) as unknown as WranglerConfig;

const d1Config = wranglerConfig.d1_databases[0];

const env = Object.fromEntries(
  ((filePath) =>
    existsSync(filePath)
      ? readFileSync(filePath, "utf-8")
          .split("\n")
          .filter((line) => line.trim() && !line.startsWith("#"))
          .map((line) =>
            line
              .split(/=(.+)/)
              .filter(Boolean)
              .map((part) => part.trim()),
          )
          .filter((parts) => parts.length === 2)
          .map(([key, value]) => [key, value?.replace(/^["']|["']$/g, "")])
      : [])(join(process.cwd(), ".dev.vars")),
) as Record<string, string>;

/*
 * NOTE: Workaround to make drizzle studio work with D1.
 * https://kevinkipp.com/blog/going-full-stack-on-astro-with-cloudflare-d1-and-drizzle/
 * Github discussion: https://github.com/drizzle-team/drizzle-orm/discussions/1545#discussioncomment-8115423
 */
export default process.env["NODE_ENV"] === "production"
  ? defineConfig({
      dialect: "sqlite",
      driver: "d1-http",
      out: "drizzle",
      schema: "src/db/schema.ts",
      dbCredentials: {
        accountId: env["CF_ACCOUNT_ID"]!,
        token: env["CF_D1_TOKEN"]!,
        databaseId: d1Config!.database_id!,
      },
    } satisfies Config)
  : defineConfig({
      dialect: "sqlite",
      out: "drizzle",
      schema: "src/db/schema.ts",
      dbCredentials: {
        url: env["DB_LOCAL_PATH"]!,
      },
    } satisfies Config);

maguayo•12/24/24, 7:03 AM

If D1 is distributed (read only) why so much diference from just a SELECT from the EU and the US?

Mmaguayo If D1 is distributed (read only) why so much diference from just a SELECT from t...

Hello, I’m Allie!•12/24/24, 7:05 AM

It’s not, D1 is single-location for now

HHello, I’m Allie!It’s not, D1 is single-location for now

maguayo•12/24/24, 7:07 AM

It's there a timeline? I am migrating everything because the documentation says it's distributed

maguayo•12/24/24, 7:09 AM

I thought you could have many instances in each region or something like that

maguayo•12/24/24, 7:10 AM

There is a whole section in D1 documentation about read replicas. Is this not true??

FFriday Candour "RETURNING *" not working on D1 how to get around that?

maguayo•12/24/24, 7:57 AM

I mean.. they market as D1 is distributed and read replicas.. I expected you know, they have read replicas.

Mmaguayo There is a whole section in D1 documentation about read replicas. Is this not tr...

Hello, I’m Allie!•12/24/24, 8:07 AM

It’s a planned feature, but it doesn’t exist yet

FFriday Candour but it's same query right?

maguayo•12/24/24, 8:14 AM

Exactly the same query from different locations.

HHello, I’m Allie!It’s a planned feature, but it doesn’t exist yet

maguayo•12/24/24, 8:15 AM

Can we know if it's planned for Q1 or more into the future?

FFriday Candour omg cloudflare

maguayo•12/24/24, 10:17 AM

Yeah.. Time to refactor everything since I counted with read replicas since their documentation mentions them...

TTrojan this happens because the API for the sqlite d1 file in miniflare is slightly dif...

theoOP•12/24/24, 6:12 PM

I might have a tool that helps a bit with managing the env/config

theoOP•12/24/24, 6:14 PM

loads config like dotenv but also has validations, and a nice wrangler integration - https://dmno.dev/

Ttheo I might have a tool that helps a bit with managing the env/config

Trojan•12/24/24, 8:38 PM

dang thanks! i’ll check it out

TTrojan dang thanks! i’ll check it out

theoOP•12/24/24, 8:44 PM

the cloudflare integration is brand new, so happy to help and work through any issues

Trojan•12/24/24, 8:52 PM

“Being able to see all of these values and which one is currently active will save you tons of headaches”

huge. im tired of “zero config” “common sense” “best practice” “no code” etc

FFriday Candour "RETURNING *" not working on D1 how to get around that?

o7•12/26/24, 11:39 PM

it works for me

import { defineConfig, type Config } from "drizzle-kit"; import { parse as parseJsonc } from "comment-json"; import { existsSync, readFileSync } from "fs"; import { join } from "path"; const wranglerConfig = parseJsonc( readFileSync(join(__dirname, "wrangler.jsonc"), "utf-8"), ) as unknown as WranglerConfig; const d1Config = wranglerConfig.d1_databases[0]; const env = Object.fromEntries( ((filePath) => existsSync(filePath) ? readFileSync(filePath, "utf-8") .split("\n") .filter((line) => line.trim() && !line.startsWith("#")) .map((line) => line .split(/=(.+)/) .filter(Boolean) .map((part) => part.trim()), ) .filter((parts) => parts.length === 2) .map(([key, value]) => [key, value?.replace(/^["']|["']$/g, "")]) : [])(join(process.cwd(), ".dev.vars")), ) as Record<string, string>; /* * NOTE: Workaround to make drizzle studio work with D1. * https://kevinkipp.com/blog/going-full-stack-on-astro-with-cloudflare-d1-and-drizzle/ * Github discussion: https://github.com/drizzle-team/drizzle-orm/discussions/1545#discussioncomment-8115423 */ export default process.env["NODE_ENV"] === "production" ? defineConfig({ dialect: "sqlite", driver: "d1-http", out: "drizzle", schema: "src/db/schema.ts", dbCredentials: { accountId: env["CF_ACCOUNT_ID"]!, token: env["CF_D1_TOKEN"]!, databaseId: d1Config!.database_id!, }, } satisfies Config) : defineConfig({ dialect: "sqlite", out: "drizzle", schema: "src/db/schema.ts", dbCredentials: { url: env["DB_LOCAL_PATH"]!, }, } satisfies Config);

I'm working on some tooling which is handling very sensitive data on behalf of my customers, along w

Similar Threads

Similar Threads

Similar Threads