Both @rozenmd and I were chatting about the state of testing (our excuse: we're in alpha). We're wor

V

vassbence•6/20/23, 2:05 PM

I think if we are able to do migrations / cleanup on the database outside of the unstable_dev() then most of the use-cases should be covered

V

vassbence•6/20/23, 2:07 PM

As for my problem I think I figured it out, by setting a preview_database_id="local" in the .toml file now I know the exact path of the db.sqlite file and thus able to execute migrations on the db before tests start running

Vvassbence I think if we are able to do migrations / cleanup on the database outside of the...

M

Matt SilverlockOP•6/20/23, 2:07 PM

Yeah, if you can run

runMigration

runMigration

runMigration

runMigration and it mirrors

wrangler d1 migrations apply <name>

wrangler d1 migrations apply <name>

wrangler d1 migrations apply <name>

wrangler d1 migrations apply <name> -> that gives you the tools you need to run a series of migrations from within CI

V

vassbence•6/20/23, 2:07 PM

that would be awesome yes

V

vassbence•6/20/23, 2:08 PM

wrangler d1 migrations apply

wrangler d1 migrations apply

wrangler d1 migrations apply

wrangler d1 migrations apply run in CI's without like logging into cloudflare? because i might just run it as a shell command as of now to make my test setup less brittle and later swithc to the programatic

runMigration

runMigration

runMigration

runMigration API once its out

Vvassbence `wrangler d1 migrations apply` run in CI's without like logging into cloudflare?...

M

Matt SilverlockOP•6/20/23, 2:16 PM

migrations apply

migrations apply

migrations apply

migrations apply defaults to remote.

Options:
      --local                 Execute commands/files against a local DB for use with wrangler dev --local  [boolean]
      --preview               Execute commands/files against a preview D1 DB  [boolean] [default: false]
      --persist-to            Specify directory to use for local persistence (you must use --local with this flag)  [string]
      --experimental-backend  Use new experimental DB backend  [boolean] [default: false]
      --batch-size            Number of queries to send in a single batch  [number] [default: 10000]

Options:
      --local                 Execute commands/files against a local DB for use with wrangler dev --local  [boolean]
      --preview               Execute commands/files against a preview D1 DB  [boolean] [default: false]
      --persist-to            Specify directory to use for local persistence (you must use --local with this flag)  [string]
      --experimental-backend  Use new experimental DB backend  [boolean] [default: false]
      --batch-size            Number of queries to send in a single batch  [number] [default: 10000]

Options:
      --local                 Execute commands/files against a local DB for use with wrangler dev --local  [boolean]
      --preview               Execute commands/files against a preview D1 DB  [boolean] [default: false]
      --persist-to            Specify directory to use for local persistence (you must use --local with this flag)  [string]
      --experimental-backend  Use new experimental DB backend  [boolean] [default: false]
      --batch-size            Number of queries to send in a single batch  [number] [default: 10000]

Options:
      --local                 Execute commands/files against a local DB for use with wrangler dev --local  [boolean]
      --preview               Execute commands/files against a preview D1 DB  [boolean] [default: false]
      --persist-to            Specify directory to use for local persistence (you must use --local with this flag)  [string]
      --experimental-backend  Use new experimental DB backend  [boolean] [default: false]
      --batch-size            Number of queries to send in a single batch  [number] [default: 10000]

V

vassbence•6/20/23, 2:17 PM

describe("backend json rest api", () => {
  const testId = randomUUID();
  let worker;

  before(async () => {
    // running migrations on db
    execSync(
      `export NO_D1_WARNING=true && wrangler d1 migrations apply db --local --persist-to=temp/${testId}`
    );

    // setting up worker
    worker = await unstable_dev("src/index.ts", {
      experimental: {
        disableExperimentalWarning: true,
      },
      persistTo: `temp/${testId}`,
    });
  });

  after(async () => {
    await worker.stop();
    execSync("rm -rf temp");
  });

  // ... tests here
});

describe("backend json rest api", () => {
  const testId = randomUUID();
  let worker;

  before(async () => {
    // running migrations on db
    execSync(
      `export NO_D1_WARNING=true && wrangler d1 migrations apply db --local --persist-to=temp/${testId}`
    );

    // setting up worker
    worker = await unstable_dev("src/index.ts", {
      experimental: {
        disableExperimentalWarning: true,
      },
      persistTo: `temp/${testId}`,
    });
  });

  after(async () => {
    await worker.stop();
    execSync("rm -rf temp");
  });

  // ... tests here
});

describe("backend json rest api", () => {
  const testId = randomUUID();
  let worker;

  before(async () => {
    // running migrations on db
    execSync(
      `export NO_D1_WARNING=true && wrangler d1 migrations apply db --local --persist-to=temp/${testId}`
    );

    // setting up worker
    worker = await unstable_dev("src/index.ts", {
      experimental: {
        disableExperimentalWarning: true,
      },
      persistTo: `temp/${testId}`,
    });
  });

  after(async () => {
    await worker.stop();
    execSync("rm -rf temp");
  });

  // ... tests here
});

describe("backend json rest api", () => {
  const testId = randomUUID();
  let worker;

  before(async () => {
    // running migrations on db
    execSync(
      `export NO_D1_WARNING=true && wrangler d1 migrations apply db --local --persist-to=temp/${testId}`
    );

    // setting up worker
    worker = await unstable_dev("src/index.ts", {
      experimental: {
        disableExperimentalWarning: true,
      },
      persistTo: `temp/${testId}`,
    });
  });

  after(async () => {
    await worker.stop();
    execSync("rm -rf temp");
  });

  // ... tests here
});

V

vassbence•6/20/23, 2:17 PM

here is my current code, looks to be working allright

V

vassbence•6/20/23, 2:18 PM

let me know if you see anything wrong

Vvassbence ``` describe("backend json rest api", () => { const testId = randomUUID(); l...

M

Matt SilverlockOP•6/20/23, 2:23 PM

Looks pretty good to me.

M

Matt SilverlockOP•6/20/23, 2:23 PM

I've stubbed out some basic docs here - https://github.com/cloudflare/cloudflare-docs/pull/9471

but need to improve with a full example.

V

vassbence•6/20/23, 2:25 PM

with the code above i do not need to use

preview_database_id

preview_database_id

preview_database_id

preview_database_id

V

vassbence•6/20/23, 2:26 PM

as simply setting the

persistTo

persistTo

persistTo

persistTo to a different temporary path works. In the docs above if an user would run tests with

preview_database_id

preview_database_id

preview_database_id

preview_database_id set but not a custom

persistTo

persistTo

persistTo

persistTo they would mess up their development database

Vvassbence with the code above i do not need to use `preview_database_id`

M

Matt SilverlockOP•6/20/23, 2:26 PM

Yeah, that makes it a bit easier in that regard. I think we need to think through how we want

preview_database_id

preview_database_id

preview_database_id

preview_database_id to interact w/ programmatic testing here.

M

Matt SilverlockOP•6/20/23, 2:26 PM

(Sometimes you might want that to happen)

V

vassbence•6/20/23, 2:27 PM

honestly what would be the perfect API for testing is to be able to have unstable_dev run D1 as in-memory only and have a function to run migrations on this in-memory sqlite db

Vvassbence honestly what would be the perfect API for testing is to be able to have unstabl...

M

Matt SilverlockOP•6/20/23, 2:30 PM

and/or we should just generate a

/tmp/<some random ID>

/tmp/<some random ID>

/tmp/<some random ID>

/tmp/<some random ID> file that gets thrown away by your OS. That way, if you have a larger DB, we don't eat all of your CI containers' memory

V

vassbence•6/20/23, 2:31 PM

sure, that would be just as good

A

amorfati•6/20/23, 8:06 PM

how can I seed local

D1

D1

D1

D1 sqlite? I was using

bun

bun

bun

bun sqlite where I create a local

db.sqlite

db.sqlite

db.sqlite

db.sqlite file and interact with it using the imported client. In the case of Cloudflare & wrangler, I'm a little confused as to how I can create and seed a local sqlite

Aamorfati how can I seed local `D1` sqlite? I was using `bun` sqlite where I create a loca...

A

amorfati•6/20/23, 8:07 PM

my

dev

dev

command

NO_D1_WARNING=true ./node_modules/.bin/wrangler dev --persist-to=./tokens.sqlite --experimental-json-config

NO_D1_WARNING=true ./node_modules/.bin/wrangler dev --persist-to=./tokens.sqlite --experimental-json-config

NO_D1_WARNING=true ./node_modules/.bin/wrangler dev --persist-to=./tokens.sqlite --experimental-json-config

NO_D1_WARNING=true ./node_modules/.bin/wrangler dev --persist-to=./tokens.sqlite --experimental-json-config

wrangler.json

wrangler.json

wrangler.json

wrangler.json

{
  "$schema": "https://gist.githubusercontent.com/o-az/d298a1776165407ba63ee763d32f7a89/raw/7f87871dd9a5efb9d3cc147fac0174ebdc9858fa/wrangler.json",
  "name": "tokens",
  "workers_dev": true,
  "main": "./src/index.ts",
  "compatibility_date": "2023-05-18",
  "compatibility_flags": [ "nodejs_compat" ],
  "placement": { "mode": "smart" },
  "d1_databases": [
    {
      "binding": "database",
      "database_name": "tokens",
      "preview_database_id": "local-tokens-sqlite"
    }
  ]
}

{
  "$schema": "https://gist.githubusercontent.com/o-az/d298a1776165407ba63ee763d32f7a89/raw/7f87871dd9a5efb9d3cc147fac0174ebdc9858fa/wrangler.json",
  "name": "tokens",
  "workers_dev": true,
  "main": "./src/index.ts",
  "compatibility_date": "2023-05-18",
  "compatibility_flags": [ "nodejs_compat" ],
  "placement": { "mode": "smart" },
  "d1_databases": [
    {
      "binding": "database",
      "database_name": "tokens",
      "preview_database_id": "local-tokens-sqlite"
    }
  ]
}

{
  "$schema": "https://gist.githubusercontent.com/o-az/d298a1776165407ba63ee763d32f7a89/raw/7f87871dd9a5efb9d3cc147fac0174ebdc9858fa/wrangler.json",
  "name": "tokens",
  "workers_dev": true,
  "main": "./src/index.ts",
  "compatibility_date": "2023-05-18",
  "compatibility_flags": [ "nodejs_compat" ],
  "placement": { "mode": "smart" },
  "d1_databases": [
    {
      "binding": "database",
      "database_name": "tokens",
      "preview_database_id": "local-tokens-sqlite"
    }
  ]
}

{
  "$schema": "https://gist.githubusercontent.com/o-az/d298a1776165407ba63ee763d32f7a89/raw/7f87871dd9a5efb9d3cc147fac0174ebdc9858fa/wrangler.json",
  "name": "tokens",
  "workers_dev": true,
  "main": "./src/index.ts",
  "compatibility_date": "2023-05-18",
  "compatibility_flags": [ "nodejs_compat" ],
  "placement": { "mode": "smart" },
  "d1_databases": [
    {
      "binding": "database",
      "database_name": "tokens",
      "preview_database_id": "local-tokens-sqlite"
    }
  ]
}

Aamorfati my `dev` command ```sh NO_D1_WARNING=true ./node_modules/.bin/wrangler dev --per...

C

Cyb3r-Jak3•6/20/23, 8:15 PM

https://developers.cloudflare.com/d1/learning/local-development/ maybe

CCyb3r-Jak3 https://developers.cloudflare.com/d1/learning/local-development/ maybe

A

amorfati•6/20/23, 8:17 PM

I've tried this. It says to use

--persist-to=/path/to/file

--persist-to=/path/to/file

--persist-to=/path/to/file

--persist-to=/path/to/file to persist data to a specific location but no luck and on seeding I couldn't find any relevant info.
seeding a local sqlite

Aamorfati I've tried this. It says to use `--persist-to=/path/to/file` to persist data to ...

C

Cyb3r-Jak3•6/20/23, 8:19 PM

What version of wrangler on you on?

A

amorfati•6/20/23, 8:21 PM

⛅️ wrangler 3.1.0 (update available 3.1.1)

⛅️ wrangler 3.1.0 (update available 3.1.1)

⛅️ wrangler 3.1.0 (update available 3.1.1)

⛅️ wrangler 3.1.0 (update available 3.1.1)

A

amorfati•6/20/23, 8:24 PM

is there a way to run a one off typescript file that seeds the db? Before wrangler I was running a

prestart

prestart

prestart

prestart script that runs a ts file to init local sqlite and seed it

V

vassbence•6/20/23, 8:32 PM

--persist-to is only a folder, so for example by using --persist-to=/temp the actual db file will be stored in /temp/v3/d1/x.sqlite

also you won't find the file right away, the db has to be accessed first for the file to be created

V

vassbence•6/20/23, 8:33 PM

take a look at

wrangler d1 migrations

wrangler d1 migrations

wrangler d1 migrations

wrangler d1 migrations, you can use a migration to seed the database or if you want a more programatic approach scroll up a bit, we had a discussion about possibilites just a few hours ago

Vvassbence take a look at `wrangler d1 migrations`, you can use a migration to seed the dat...

A

amorfati•6/20/23, 8:35 PM

ok will try scrolling up. My attempt was to make seeding a worker endpoint that I call by hitting the endpoint but will see if I find a better approach

Vvassbence --persist-to is only a folder, so for example by using --persist-to=/temp the ac...

A

amorfati•6/20/23, 8:37 PM

ah gotcha. The wording in the doc made me think I can specify filename

--persist-to=/path/to/file

--persist-to=/path/to/file

--persist-to=/path/to/file

--persist-to=/path/to/file

Aamorfati ok will try scrolling up. My attempt was to make seeding a worker endpoint that ...

M

Matt SilverlockOP•6/20/23, 9:10 PM

https://developers.cloudflare.com/d1/learning/local-development/#test-programmatically (specifically that section)

M

Matt SilverlockOP•6/20/23, 9:11 PM

If you set

preview_database_id

preview_database_id

preview_database_id

preview_database_id (per https://developers.cloudflare.com/d1/platform/migrations/#wrangler-customizations) and use

wrangler d1 migrations apply <db-name> --local

wrangler d1 migrations apply <db-name> --local

wrangler d1 migrations apply <db-name> --local

wrangler d1 migrations apply <db-name> --local -> you can apply migrations locally to seed the DB. As above, you can also do that programatically via your test harness

Migrations · Cloudflare D1 docs

Database migrations are a way of versioning your database. Each migration is stored as an .sql file in your migrations folder. The migrations folder …

MMatt Silverlock If you set `preview_database_id` (per https://developers.cloudflare.com/d1/platf...

A

amorfati•6/20/23, 9:15 PM

I don't see the db client to interact with sqlite being used in the section or any sql queries to seed being done. I assume this section expects me to run migrations through cli first, but what if I want to seed programmatically

Aamorfati I don't see the db client to interact with sqlite being used in the section or a...

A

amorfati•6/20/23, 9:16 PM

is it possible to tell wrangler to use a specific

name.sqlite

name.sqlite

name.sqlite

name.sqlite file?

A

amorfati•6/20/23, 9:16 PM

Thank you for the help by the way

Original message was deleted

A

amorfati•6/20/23, 9:23 PM

that can run a ts file and pass it the context with the database client? I will try that

A

amorfati•6/20/23, 9:23 PM

looks promising

A

amorfati•6/20/23, 9:35 PM

ok fair. That should work too

Aamorfati I don't see the db client to interact with sqlite being used in the section or a...

M

Matt SilverlockOP•6/20/23, 9:45 PM

You can invoke

wrangler d1 migrations

wrangler d1 migrations

wrangler d1 migrations

wrangler d1 migrations inside your tests: the example shows that.

Aamorfati I don't see the db client to interact with sqlite being used in the section or a...

M

Matt SilverlockOP•6/20/23, 9:46 PM

You can also read our discussion earlier: d1-database

MMatt Silverlock You can also read our discussion earlier: https://discord.com/channels/595317990...

A

amorfati•6/20/23, 10:32 PM

sweet, thank you

A

amorfati•6/20/23, 10:33 PM

when I created a new local db just now, it said binding is

db

db

db

db. Is there a way to change that by just changing

binding

binding

binding

binding in wrangler config?

D

Dahoom•6/20/23, 10:57 PM

use the --local option

D

Dahoom•6/20/23, 10:58 PM

so something like

wrangler d1 execute --local --comand="SELECT NOTHING"

wrangler d1 execute --local --comand="SELECT NOTHING"

wrangler d1 execute --local --comand="SELECT NOTHING"

wrangler d1 execute --local --comand="SELECT NOTHING"

DDahoom so something like ``wrangler d1 execute --local --comand="SELECT NOTHING"``

D

Dahoom•6/20/23, 10:58 PM

D

Dahoom•6/20/23, 11:04 PM

you can use wrangler

D

Dahoom•6/20/23, 11:04 PM

i tested it and it works in my pages directory

D

Dahoom•6/20/23, 11:05 PM

even when running wrangler pages dev it will read the wrangler.toml file and load the DB inside it

Both @rozenmd and I were chatting about the state of testing (our excuse: we're in alpha). We're wor

Similar Threads