Cloudflare Developers•14mo ago

Not really a client it's actual full SQLite

Isaac McFadyenOP•11/21/24, 8:37 PM

Oh yeah sql.js is also one

elpupper•11/21/24, 8:38 PM

https://github.com/sql-js/sql.js

elpupper•11/21/24, 8:38 PM

am i missing something about wasm?

elpupper•11/21/24, 8:38 PM

why wasm in the backend?

IIsaac McFadyen Not really a client it's actual full SQLite

elpupper•11/21/24, 8:40 PM

same with the one i sent i believe

Eelpupper why wasm in the backend?

Isaac McFadyenOP•11/21/24, 8:51 PM

If you are building a native application in JavaScript (using Electron for instance), or are working in node.js, you will likely prefer to use a native binding of SQLite to JavaScript.

They recommend against that actually

IIsaac McFadyen > If you are building a native application in JavaScript (using Electron for ins...

elpupper•11/21/24, 9:00 PM

Talking about cloudflare in general :NotLikeThis:

Eelpupper Talking about cloudflare in general :NotLikeThis:

Isaac McFadyenOP•11/21/24, 9:02 PM

Mainly because WASM is portable and secure. If you allowed anyone to upload binaries (as does AWS Lambda/GCP Cloud Run) then to get enough security you basically need to run them in a full, dedicated VM or at least gVisor (GCP). With WASM even though it's untrusted code, running inside the V8 WASM sandbox is secure enough.

Isaac McFadyenOP•11/21/24, 9:03 PM

And because it's running in V8 it's far lighter-weight than a VM/gVisor and they can afford to run thousands/tens of thousands of WASM sandboxes.

Isaac McFadyenOP•11/21/24, 9:05 PM

There's a good writeup here: https://developers.cloudflare.com/workers/reference/security-model/

Max (@rozenmd)•11/21/24, 9:27 PM

D1s alpha was SQLite in WASM on DOs, it was uh "fun" to operate

MMax (@rozenmd)D1s alpha was SQLite in WASM on DOs, it was uh "fun" to operate

elpupper•11/21/24, 10:42 PM

:kekw:

IIsaac McFadyen And because it's running in V8 it's far lighter-weight than a VM/gVisor and they...

elpupper•11/21/24, 10:42 PM

i will have to test this

elpupper•11/21/24, 10:43 PM

im working o nsomething similar to AWS lambda and CF workers using NixOS microvm.nix and firecracker

JJames Already use straight SQL, there's no getting around some big requests with lots ...

Burrito•11/21/24, 11:58 PM

Worker smart placement should help with this though, as it should put your worker closer to D1 rather than closer to user if it sees that you are making multiple round trips to D1. Read replica is also on the roadmap, although there doesn't seem to be any ETA for that.

Burrito•11/22/24, 12:04 AM

I'm also considering if I should build on D1 or DO + SQLite storage, but the latter having to pay extra for DO requests, as well as losing out on D1 features, are putting me off.

BBurrito I'm also considering if I should build on D1 or DO + SQLite storage, but the lat...

James•11/22/24, 12:13 AM

Had the same concern, what clinched it for me is the 10gb limit on d1. Audit trails, request logging etc eats that up quick. Only d1 feature I miss is dumping the SQLite file. Going custom on the rest isn’t hard.

Burrito•11/22/24, 12:15 AM

10 GB limit? Ah, maximum database size.

JJames Had the same concern, what clinched it for me is the 10gb limit on d1. Audit tra...

elpupper•11/22/24, 12:32 AM

make new dbs i guess

elpupper•11/22/24, 12:33 AM

but i swear cloudflare already does those for you

JJames Had the same concern, what clinched it for me is the 10gb limit on d1. Audit tra...

Burrito•11/22/24, 12:45 AM

DO SQLite currently only has 1 GB limit though, and doc seems to indicate that it will be raised to 10 GB when GA, which matches what D1 offers.

BBurrito DO SQLite currently only has 1 GB limit though, and doc seems to indicate that i...

James•11/22/24, 1:25 AM

1gb per user is reasonable for a multi tenant app. Can scale that to terabytes if needed. Much harder to scale d1 to 1tb.

SteffTek•11/22/24, 8:41 AM

Well i truly think my worker smart placement is bugged, it wont analyze anything anymore, not even stating something in the dashboard. has the minimum traffic for smart placement increased in someway?

SSteffTek Well i truly think my worker smart placement is bugged, it wont analyze anything...

Walshy•11/22/24, 9:04 AM

could you send me your account id please?

WWalshy could you send me your account id please?

SteffTek•11/22/24, 9:05 AM

Sure! Account ID is f61e2912c0937987fbd186d3523dc847 and our Dev Worker is called "worker-customer-devinc" (screenshot above is from one of our tenants)

Walshy•11/22/24, 9:05 AM

Thanks let me take a look

Walshy•11/22/24, 9:14 AM

So, it looks like smart placement was re-enabled 30 mins ago and yeah, so far it doesn't have enough requests

WWalshy So, it looks like smart placement was re-enabled 30 mins ago and yeah, so far it...

SteffTek•11/22/24, 9:14 AM

yeah i did it (a few times in fact) to see if it reappears

SteffTek•11/22/24, 9:14 AM

the other worker hadn't hat it reenabled and doesnt have it either

Walshy•11/22/24, 9:15 AM

I can check that one too, what's it called?

SteffTek•11/22/24, 9:15 AM

worker-customer-***

SteffTek•11/22/24, 9:15 AM

last deployment there was a month ago

Walshy•11/22/24, 9:18 AM

It looks like that also has pretty minimal traffic. looks like it was placed on the 13th and then not on the 15th - let me see what changed

SteffTek•11/22/24, 9:19 AM

Thx! The problem is each of our tenants won't have much traffic. we are a cloudbased cash register provider, so we only have a limited amount of users per tenant

Walshy•11/22/24, 9:21 AM

Ok I see what changed.. figuring out why

Walshy•11/22/24, 9:25 AM

Ok yep, looks like an issue our side - gonna let the team know!
Thanks for reporting, we'll get it fixed!

WWalshy Ok yep, looks like an issue our side - gonna let the team know! Thanks for repor...

SteffTek•11/22/24, 9:25 AM

<3 Thank you!!!

Kimitri•11/22/24, 9:43 AM

can someone help me with this. I'm trying to connect vitest with D1 local generated by wrangler dev, but it's not working, I use Vitest-pool-workers to do it with this configuration:

import { defineWorkersConfig } from "@cloudflare/vitest-pool-workers/config";

export default defineWorkersConfig({
  
  test: {
    globals: true,
    poolOptions: {
      workers: {
        isolatedStorage: true,
        wrangler: { configPath: "./wrangler.toml" },
      },
    },
  },
});

import { defineWorkersConfig } from "@cloudflare/vitest-pool-workers/config";

export default defineWorkersConfig({
  
  test: {
    globals: true,
    poolOptions: {
      workers: {
        isolatedStorage: true,
        wrangler: { configPath: "./wrangler.toml" },
      },
    },
  },
});

and to ensure I tried to show in the logs like this

import { Hono } from 'hono'

export type Env = {
  DB: D1Database
}
const app = new Hono<{ Bindings: Env}>()

app.get('/', async (c) => {

  console.log(c.env.DB)

  return c.text("Hello world!")
})

export default app

import { Hono } from 'hono'

export type Env = {
  DB: D1Database
}
const app = new Hono<{ Bindings: Env}>()

app.get('/', async (c) => {

  console.log(c.env.DB)

  return c.text("Hello world!")
})

export default app

and it's show undefined, it just show some data when I run wrangler dev,

andifined.dev•11/22/24, 10:33 AM

Is it possible to increase the 10 gb Limit d1 has currently?

Aandifined.dev Is it possible to increase the 10 gb Limit d1 has currently?

1984 Ford Laser•11/22/24, 10:39 AM

In the docs: https://developers.cloudflare.com/d1/platform/limits/

Cloudflare Docs

Limits | Cloudflare D1 docs

Cloudflare also offers other storage solutions such as Workers KV, Durable Objects, and R2. Each product has different advantages and limits. Refer to Choose a data or storage product to review which storage option is right for your use case.

andifined.dev•11/22/24, 10:40 AM

Yes, already read that, but was not sure if applicable to the db limit in General too. Thanks!

Aandifined.dev Yes, already read that, but was not sure if applicable to the db limit in Genera...

1984 Ford Laser•11/22/24, 10:47 AM

I think most of the limits can be increased, its mostly about being able to justify your use case. If you need the 10GB DB limit raised, you would probably want the 250GB account limit raised too?

andifined.dev•11/22/24, 10:48 AM

Don‘t know yet, just exploring possibilities. And if there’s no hard cap that can’t be raised, d1 seems good to go

Aandifined.dev Don‘t know yet, just exploring possibilities. And if there’s no hard cap that ca...

1984 Ford Laser•11/22/24, 10:49 AM

If you are storing close top 10GB in a single DB, there is probably a better way to store/split the data up. If you do ever need the limit increased, I'd be asking way in advance

Wes•11/22/24, 9:37 PM

Hi all. Does anyone know/heard if there are any plans to run D1 in samsam? It just says it's not available but no comments on weather there are plans or not. (https://developers.cloudflare.com/d1/configuration/data-location/#available-location-hints)

SSteffTek <3 Thank you!!!

Walshy•11/23/24, 1:13 AM

Heyo, sorry for not getting back sooner - just lots of other stuff going on

Eng released a fix and I can see your Workers being placed again

WWalshy Heyo, sorry for not getting back sooner - just lots of other stuff going on Eng...

SteffTek•11/23/24, 9:04 AM

Thx! I can confirm for our test tenant, but not the prod one. Still says insufficient invocations

after testing, the responsetime of our test tenant is back to the original ^^

Kotkoroid•11/23/24, 9:21 AM

Hey, where do I find D1 repository? Couldn't find it on GitHub.

James•11/23/24, 4:17 PM

For what specifically? You'll find some of D1 source in the workerdworkerd repo like https://github.com/cloudflare/workerd/blob/main/src/workerd/util/sqlite.c%2B%2B, but if you're new to D1, the Getting Started guide is going to the best place to start: https://developers.cloudflare.com/d1/get-started/

Cloudflare Docs

Get started | Cloudflare D1 docs

This guide instructs you through:

TOPGUN•11/23/24, 7:40 PM

Hello everyone,

Quick question. Is D1 a suitable backend database for a financial wallet web application? i.e. ensuring consistency across users funds balances / withdrawing funds and depositing funds reflect consistently across all users of the wallet app. Is D1 sufficient for this use case? or will i need a Postgres db for example?

Thanks,

Not really a client it's actual full SQLite

Similar Threads