Thank you for open sourcing this, it's unbelievably helpful
Thank you for open sourcing this, it's unbelievably helpful
KFWIW, SQLite is designed to be secure, as it is used extensively in many places where such security is needed. For instance, Google Chrome has long given arbitrary web sites the ability to store data in sqlite databases via the WebSQL API (although this API has been deprecated due to inability to achieve standardization). The library is extensively fuzzed. There have been some security vulnerabilities over the years, but far fewer than, say, V8. You should of course carefully read https://www.sqlite.org/security.html .
KChanges land in workerd first; it's production that lags behind. 
Pdid google employ any sort of isolation techniques non the less that you are aware of?
KI'm not super familiar with the details but I would assume they ran sqlite inside the renderer process, which is heavily sandboxed
Kthat is a second layer of protection, intended as defense in depth against vulnerabilities
Pbecause the worst fear is that, should there be some undiscovered issue, and a user gets arbitrary command access, they could read other people's databases which is terrifying. So somehow I need to figure out how to give each database it's own sandbox, and that's frankly where I run out of knowledge. It would be cool to learn, but if I mess up here that's a HUGE issue. Maybe this isn't even worth attempting. It would be incredibly cool, but maybe it isn't worth attempting
PI guess the context is that I'm making a personal analytics product. Kinda sorta grafana esq or if you've seen datasette like that. So it feels natural to give each person their own database and visual tools to play with it, but also ways to directly query it, but if you have a vulnerability well thats a lot of sensitive data
Kbuilding sandboxes is a bit of a niche skill. You could use something off-the-shelf like firecracker or gvisor. For this purpose, you might also be OK using basic containers (if configured correctly), or writing something custom based on Linux namespaces and seccomp (not actually that hard once you figure it out).
PI'll look at the things you listed. Containers for each user feels very expensive, but the other ones are potentially excellent
KHonestly though, the rate of critical vulnerabilities in SQLite is like once every few years. If you're on the ball about patching you might not need per-user layer 2 sandboxing.
PI'm also considering just using D1 and giving each user their own database to scale until I can hopefully afford to pay someone to do it for me
PI feel like I recall reading somewhere this is a supported use case
Mit is, just the UX needs a bit of work
Pyeah I was thinking that the prospect of having thousands of these is scary
Mthe eventual idea is to have a DO-style binding where you don't need to call the API to patch your worker's bindings for new dbs
Pfrom a visual standpoint, ideally I imagine having like "groups" and then a group contains databases
Pand the group is presented more like a conventional sql interface (tight rows)
Poh... like KV is!
Pand then like this might become or (because while I'm not familiar with sandboxing, I am very familiar with abusing proxy objects to do terrible things in typescript)
Moh yeah the D1 UI dates back to when we thought 10 dbs per account was a lot, we'll be refreshing it
PI dunno I just think that one more level of nesting is appropriate here. Like on my personal account I have tons of projects, and I wouldn't want one project crowding the interface that I use for other projects with hundreds of databases. Maybe the suggestion here is make a... is it a zone? organization? account?
CWhat extensions are you looking for? The list of allowed is at https://github.com/cloudflare/workerd/blob/132c62a3916c2ad5413b2a3efd9a9b7a0242532c/src/workerd/util/sqlite.c%2B%2B#L126
GitHub
The JavaScript / Wasm runtime that powers Cloudflare Workers - cloudflare/workerd
Pprobably SpatiaLite but if D1 supports R*Tree you'd get by fine
Wdoes D1 support locking?
USQLite is fine for a lot of users though you may need read replicas and sharding as with literally any database at some scale.
D1 doesn’t have the built in methods for this yet but will come before production ready.
If you want something production ready right now durable objects is the best option to stay on cloudflare.
D1 doesn’t have the built in methods for this yet but will come before production ready.
If you want something production ready right now durable objects is the best option to stay on cloudflare.
WI rely heavily on the d1 http api in my postgres app. I am trying to use d1 as an edge cache to postgres so that cloudflare workers can connect directly to d1 instead of postgres reducing latency. In order to do that I added an insert trigger in postgres to call the
D1 api sometimes fails randomly with no clear reason. I have implemented a queue system to collect all of the failed calls and here are some error msgs I get
d1 http apiD1 api sometimes fails randomly with no clear reason. I have implemented a queue system to collect all of the failed calls and here are some error msgs I get
(Strange, as I do not change authentication and only some calls fail with this error{"content":"{\"success\":false,\"errors\":[{\"code\":10000,\"message\":\"Authentication error\"}]}\n","error_msg":null,"status_code":500}{"content":"{\n \"result\": null,\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7501,\n \"message\": \"Internal error: D1 API returned: (503 Service Unavailable) [truncate html, see screenshot]{"content":"{\n \"result\": null,\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7502,\n \"message\": \"Internal error: db error: FATAL: server conn crashed?\"\n }\n ],\n \"messages\": []\n}","error_msg":null,"status_code":500}{"content":"{\n \"result\": [],\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7500,\n \"message\": \"Cannot resolve D1 due to transient issue on remote node.\"\n }\n ],\n \"messages\": []\n}","error_msg":null,"status_code":500}{"content":"{\n \"result\": null,\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7500,\n \"message\": \"Internal error: Flipper failed: Flipper API returned: error sending request for url (https://flipper.sd.cfplat.com/features/edgeworker?acct=8640544): error trying to connect: dns error: failed to lookup address information: Name or service not known\"\n }\n ],\n \"messages\": []\n}","error_msg":null,"status_code":500}
Interact with Cloudflare's products and services via the Cloudflare API
W- `{"content":null,"error_msg":"Timeout was reached","status_code":null}` - this is my app timing out the HTTP request. I am not sure how long the timeout is, but it's likely around 5+ seconds.
{"content":"upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination","error_msg":null,"status_code":503}{"content":"{\n \"result\": null,\n \"success\": false,\n \"errors\": [\n {\n \"code\": 7501,\n \"message\": \"Internal error: D1 API returned: (409 Conflict) error code: 1018\"\n }\n ],\n \"messages\": []\n}","error_msg":null,"status_code":500}{"content":"<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n","error_msg":null,"status_code":502}
KThe HTTP API at api.cloudflare.com is intended to be used for configuration and debugging, not live serving. It won't have anywhere near the availability and performance of the D1 API called from a Worker. If you want HTTP access to your D1 database that can be used as part of a production stack, I would recommend writing a Worker that implements an HTTP API backed by a D1 binding, rather than using api.cloudflare.com.
WLet’s say I made 90 database, each database is to hold one day worth of data, this is to circumvent the database size limit
Then I bind all of them to my worker
How can I tell which db to use to my worker?
Let’s say the if input is 2024-01-25 then use env.db_2024-01-25
I am struggling to use user input as variable name
Then I bind all of them to my worker
How can I tell which db to use to my worker?
Let’s say the if input is 2024-01-25 then use env.db_2024-01-25
I am struggling to use user input as variable name
Vis there a timeline for d1? when could we expect a stable release?
Vi want to use it in prod
Vik know its a beta, but would like to see how it stacks up against another sqlite service, like Turso
FHello everyone,
I'm trying to deploy a worker with a D1 DB and I'm getting this error: (what am I doing wrong?)
I'm trying to deploy a worker with a D1 DB and I'm getting this error: (what am I doing wrong?)
KYou need to use Module Workers (
export defaultaddEventListener
FThank you @kian
Mhttps://blog.cloudflare.com/d1-open-beta-is-here/
We’re setting a clear goal: we want to make D1 “generally available” (GA) for production use-cases by early next year (Q1 2024)The end of Q1 is April, so around then
MA fix for that is in-progress by @geelen
YIs there a way to tell wrangler where to store the db locally?
I have a monorepo with CF pages and a worker both using the same DB.
Using miniflare on the pages app to do this. I'm guessing I should just do the same for the worker?
I have a monorepo with CF pages and a worker both using the same DB.
Using miniflare on the pages app to do this. I'm guessing I should just do the same for the worker?
BYou're looking for "--persist-to" I believe
YAha! Totally missed that, thanks!
TThis just made me monkey tilt. It's not even listed on the d1 limits page.
The SQLite default is 500 btw.
EDIT: just found out SQLite has window functions
The SQLite default is 500 btw.
EDIT: just found out SQLite has window functions
K3 is the recommended high-security value by SQLite.
K
MHi Everyone
Can anyone help me to connect D1 database to mySql workbench?
Can anyone help me to connect D1 database to mySql workbench?
MD1 is SQLite over HTTP, not MySQL - it might work with tools that support custom JDBC drivers: https://github.com/isaac-mcfadyen/d1-jdbc-driver
Mhttps://github.com/isaac-mcfadyen/d1-jdbc-driver/blob/main/src/main/java/org/isaacmcfadyen/D1Queryable.java#L80 surprised this endpoint hasn't changed in 2 years