When D1 is released I assume it will have much much much higher limits to storage than it does now.

When D1 is released I assume it will have much much much higher limits to storage than it does now. But it is going to be in alpha for some time so you shouldnt plan on using it if you are already working on a service you want to use for critical production workloads

UUnsmart When D1 is released I assume it will have much much much higher limits to storag...

MetalGear•6/18/23, 7:00 PM

Thanks I get that and using planet scale for now but watching d1 carefully as I’d like to migrate to using only cf services in future so that’s why I’m asking

MetalGear•6/18/23, 7:00 PM

My question I guess is does cloudflare plan to auto scale for users like planet scale does

MetalGear•6/18/23, 7:01 PM

Or is that something they have settled that we have to do ourselves with sharding? How would one even handle having multiple databases due to reaching size limits

JSON•6/19/23, 3:22 AM

I just saw that new information was updated in d1 price.
The price of reading is reasonable
It may not be completed yet, but I have a few questions.

Does the UPDATE query go into Write units?
Does "Workers Paid" mean First 50 million is free? Or are you saying that up to 50 million is charged $1 per million?

UnsmartOP•6/19/23, 3:25 AM

Updates are writes.

For workers paid plan the first 50 million write units are free. The first 25 billion read units are free. (But pricing isn’t final and subject to change)

UUnsmart Updates are writes. For workers paid plan the first 50 million write units are...

JSON•6/19/23, 3:28 AM

Thank you for your answer!
Then I think it's much cheaper than I expected!
I will change my postgres db to sqlite sometime this week to move quickly when d1 is released!

JSON•6/19/23, 3:31 AM

I have one last question

When using the Workers Paid model
Do I get 5 million every day in the free section in addition to the first 25 billion?

UnsmartOP•6/19/23, 3:38 AM

No those tiers are separate

UUnsmart No those tiers are separate

JSON•6/19/23, 3:48 AM

Thank you. I had no conscience lol

Ggeelen Our underlying infrastructure is quite different to PlanetScale, which makes cer...

Steved•6/19/23, 8:59 AM

Do we know yet how multiple databases will work yet? At the moment the limitations of the binding make having multiple DBs awkward. I'm hoping to use a separate DB per SaaS tenant. I can't currently see how that would fit the current bindings. Other systems, e.g. Azure SQL have a logical 'Server' that contains individual databases. That would work (assuming you don't call it 'Server').

SSteved Do we know yet how multiple databases will work yet? At the moment the limitatio...

Matt Silverlock•6/19/23, 12:47 PM

We have some ideas here for the “thousands to tens of thousands of tenants” - that will isolate each DB you create. Can’t share more yet and won’t be “next month”.

Jjoostschuur Can I access a binding to use D1 from a local Node script (i.e. not a worker whe...

Matt Silverlock•6/20/23, 10:51 AM

Your current options:

Shell out to
```
wrangler
```
```
wrangler
```
and have it run commands/import files for you
Put your own HTTP API in front of D1
Use
```
wrangler dev
```
```
wrangler dev
```
in remote mode against your live DB

Jjoostschuur To be honest, I am kind of tempted to just bundle all my data right into the wor...

Matt Silverlock•6/20/23, 11:15 AM

Why not KV? If it’s read heavy. You can bundle into your script but will eventually hit script size limits.

vassbence•6/20/23, 11:22 AM

Hi!

I've spent quite a few hours looking up this specific use-case but could not find much help online so I'd like to ask:

How can I work with D1 in the context of integration/e2e tests?
Let's say I have a JSON REST API written in a Worker.

As for the Worker itself I can use wranglers unstable_dev()unstable_dev() API to set up everything for testing.

My code currently looks like this:

describe("backend json rest api", () => {
  let worker;

  before(async () => {
    worker = await unstable_dev("src/index.ts", {
      experimental: { disableExperimentalWarning: true },
    });
  });

  after(async () => {
    await worker.stop();
  });

  // ... actual tests here
});

describe("backend json rest api", () => {
  let worker;

  before(async () => {
    worker = await unstable_dev("src/index.ts", {
      experimental: { disableExperimentalWarning: true },
    });
  });

  after(async () => {
    await worker.stop();
  });

  // ... actual tests here
});

At the very least it would be nice to have a fresh database ready at the start of the test. So like create a new SQLite file and run all the migrations. Currently the worker in the code above simply accesses my local D1 file.

How could I accomplish this in a way that would later fit into CI/CD etc?

Jjoostschuur Let me see if I get this right: If I wanted to trigger a sync process between KV...

vassbence•6/20/23, 11:26 AM

There are Unbound Workers which have no execution time limit.

Vvassbence Hi! I've spent quite a few hours looking up this specific use-case but could n...

vassbence•6/20/23, 11:32 AM

As for my usecase I guess I could run shell scripts before testing to clean up and migrate the database but it would be nice if the tests somehow had their own instance/database file and they would not interfere with the local development file.

Any insight?

vassbence•6/20/23, 11:37 AM

persistTo to a temporary file and execute migrations in the before function?

vassbence•6/20/23, 11:39 AM

AFAIK wrangler d1 migrations appy just runs the migration files one after another with execute()

vassbence•6/20/23, 11:39 AM

and so I can do the same without using child_process or similar node apis to keep everything progamatical in the test

vassbence•6/20/23, 11:44 AM

oh yeah my bad you can not access the actual D1 APIs from the test, so I either have to create an endpoint inside the Worker to run the migrations or fall back to child_process.exec to run wrangler d1 migrations apply

vassbence•6/20/23, 11:47 AM

Guess I just wanted to have one less dependency.. So you run better-sqlite3 on the db file which gets generated in a known location (since we set it in persistTopersistTo) then read the migrations files and execute them via better-sliqte3?

vassbence•6/20/23, 11:56 AM

Wrangler usees Miniflare as per the codebase but then I guess miniflare 3 is using better-sqlite3?

Max (@rozenmd)•6/20/23, 12:03 PM

IIRC it might even be in the codebase twice

vassbence•6/20/23, 12:07 PM

I guess this is not high on the roadmap but do you think this developer experience around testing D1 will improve later on?

Max (@rozenmd)•6/20/23, 12:09 PM

d1's dev experience in general has a long way to go, yeah

vassbence•6/20/23, 12:11 PM

but hey, the product is already super awesome

i've been waiting years for a truly serverless database. it's even better that it's still running good old SQL and there is not yet another NoSQL API to learn

vassbence•6/20/23, 12:15 PM

Later down the road I'd love to learn more about the limitations of the DB. To given an example DynamoDB is very limited in some ways but they are very exact about the limitations and also clearly state that why these limitations are in place (e.g.: no joins to have predicatable response times even with TBs of data).

Regular SQL databases inherently have a lot of complexity and are much more of a black-box in this sense but it would be still nice to learn about the limits of the platform and also maybe about any changes you guys made and more importantly why those changes were made (eg.: disable transactions because of X to get better Y metric)

Jjoostschuur Let me see if I get this right: If I wanted to trigger a sync process between KV...

Matt Silverlock•6/20/23, 12:16 PM

Ask in #kv - but to correct some things:

The execution time is CPU time. Writing / reading from data storage isn’t counted.
Unbound is part of the paid tier. Not a beta!
KV also has a REST API you can bulk write to. See the KV docs.

vassbence•6/20/23, 12:19 PM

Also pardon me if it's not appropriate to ask but I'd love to know what the vision is for D1. Will it be truly a fully fledged SQL database that supports all and every use case that we can currently use a regular run of the mill sql database for? Or it will just focus on being good for cloud use cases and because of this maybe cut back in some other parts and not support some usecases?

Vvassbence Also pardon me if it's not appropriate to ask but I'd love to know what the visi...

Matt Silverlock•6/20/23, 12:37 PM

https://blog.cloudflare.com/d1-turning-it-up-to-11/ (and older blogs). I don’t think “cut back” and “fully fledged” make sense - every database has inherent limitations. AWS RDS can scale up, but you have to plan ahead, or provision capacity units, and it’s strictly regional. Is that fully fledged or also limited?

The Cloudflare Blog

D1: We turned it up to 11

We've been heads down iterating on D1, and we've just shipped a major new version that's substantially faster, more reliable, and introduces Time Travel: the ability to restore a D1 database to any point in time.

itsmatteomanf•6/20/23, 12:46 PM

One question I have regarding optimizing the actual code.
If you do a env.DB.prepare()env.DB.prepare() at the start of the file for 10 will it incur a performance penalty/go to the actual DB?
I doubt it, but I also expect it might be slighty slower that doing it just when needed, if that statement is maybe used only once?

vassbence•6/20/23, 1:03 PM

env.DB.prepare() does not send any type of request IIRC, you have to explicitly add .all() or .first() to send it to the DB

kian•6/20/23, 1:04 PM

Yeah it just constructs a D1PreparedStatementD1PreparedStatement object

vassbence•6/20/23, 1:25 PM

@skye_31 persistTo only sets the folder in which D1 will be stored but somehow I still need to figure out the uuid of the database, how do you do that?

vassbence•6/20/23, 1:26 PM

settings persistTopersistTo to like testtest will place the db into test/v3/d1/{uuid}test/v3/d1/{uuid} where uuid is the id of the database in the .toml file. Not sure how should I access that in the test

vassbence•6/20/23, 1:27 PM

in like a non-hacky way

Vvassbence env.DB.prepare() does not send any type of request IIRC, you have to explicitly ...

itsmatteomanf•6/20/23, 1:46 PM

Yeah, Iwas sure about no queries, but maybe...

When D1 is released I assume it will have much much much higher limits to storage than it does now.

Similar Threads

Similar Threads

Similar Threads