I just noticed a table called `_cf_KV` on every D1 database. This table has two columns, `key` and `

Jjuri I just noticed a table called `_cf_KV` on every D1 database. This table has two ...

H

That might be prep for the SQL-on-DOs thing they are coming out with soon. According to Kenton(https://canary.discord.com/channels/595317990191398933/773219443911819284/1128025139632681131), regular DO KV storage will be stored in a SQLite Table, which might be what that is...

J

juriOP•7/11/23, 3:55 PM

Thanks for the insights, that's very interesting and definitely good do know!

Jjuri I just noticed a table called `_cf_KV` on every D1 database. This table has two ...

M

Matt Silverlock•7/11/23, 4:35 PM

Nothing to do with KV the product

M

micromashor•7/11/23, 4:46 PM

is there a size limit to what you can do with

db.batch()

db.batch()

db.batch()

db.batch() and/or

db.exec()

db.exec()

db.exec()

db.exec()?

B

Braden•7/11/23, 8:02 PM

It appears

npx wrangler d1 migrations apply some-db --experimental-backend

npx wrangler d1 migrations apply some-db --experimental-backend

npx wrangler d1 migrations apply some-db --experimental-backend

npx wrangler d1 migrations apply some-db --experimental-backend only works when logged in via

npx wrangler login

npx wrangler login

npx wrangler login

npx wrangler login. In CI, however, where I'm not running

npx wrangler login

npx wrangler login

npx wrangler login

npx wrangler login but instead providing a

CLOUDFLARE_API_TOKEN

CLOUDFLARE_API_TOKEN

CLOUDFLARE_API_TOKEN

CLOUDFLARE_API_TOKEN env variable the command exits with code 1. Is this a known issue? What workarounds exist for running migrations in CI?

B

Braden•7/11/23, 8:05 PM

(I've confirmed I can run commands such as

npx wrangler pages deploy dist --project-name abc

npx wrangler pages deploy dist --project-name abc

npx wrangler pages deploy dist --project-name abc

npx wrangler pages deploy dist --project-name abc in CI only providing the

CLOUDFLARE_API_TOKEN

CLOUDFLARE_API_TOKEN

CLOUDFLARE_API_TOKEN

CLOUDFLARE_API_TOKEN env var.)

B

Braden•7/11/23, 8:30 PM

Yeah, also provided that. Unfortunately I don't see any additional output.

B

Braden•7/11/23, 8:31 PM

Since it works locally with

wrangler login

wrangler login

wrangler login

wrangler login I assume this is probably just an auth bug? Is there any way to provide auth in CI besides using the env vars?

B

Braden•7/11/23, 8:31 PM

i.e. could I somehow make

wrangler login

wrangler login

wrangler login

wrangler login work in CI?

B

Braden•7/11/23, 8:33 PM

Good question. I'll look into that. Thanks!

B

Braden•7/11/23, 8:36 PM

That was it!! Thanks @skye_31

B

Braden•7/11/23, 8:37 PM

Didn't have D1 permissions on the token. Appreciate the help!

TTechist Created a D1 database locally and tested it. Now trying to move the schema and ...

M

Max (@rozenmd)•7/12/23, 7:39 AM

your wrangler.toml needs to have a

main

main

entry pointing at your worker entrypoint, like

main = "src/index.ts"

main = "src/index.ts"

main = "src/index.ts"

main = "src/index.ts"

TTechist Created a D1 database locally and tested it. Now trying to move the schema and ...

M

Matt Silverlock•7/12/23, 10:36 AM

Did you create the DB on its own? Strongly suggest following the guide at least once: https://developers.cloudflare.com/d1/get-started/

T

theLaCzar•7/12/23, 12:32 PM

had anyone setup serverless framework with D1?

TtheLaCzar had anyone setup serverless framework with D1?

M

Matt Silverlock•7/12/23, 1:20 PM

Which specific framework? We have a ton of examples:

D1 + Hono: https://developers.cloudflare.com/d1/tutorials/build-a-comments-api/
Full stack framework code snippets: https://developers.cloudflare.com/d1/examples/
Long list of framework guides: https://developers.cloudflare.com/pages/framework-guides/

T

theLaCzar•7/12/23, 2:12 PM

Serverless Framework https://www.serverless.com/. im using that to build by cf workers

Serverless: Develop & Monitor Apps On AWS Lambda

Easily develop and monitor auto-scaling applications on AWS Lambda, API Gateway, DynamoDB, etc., with the Serverless Framework and Serverless Monitoring Dashboard.

TtheLaCzar Serverless Framework https://www.serverless.com/. im using that to build by cf w...

M

Max (@rozenmd)•7/12/23, 3:12 PM

https://github.com/cloudflare/serverless-cloudflare-workers

last commit on Mar 26, 2019

ouch

M

Max (@rozenmd)•7/12/23, 3:13 PM

I don't think that serverless plugin supports wrangler 2, let alone D1

T

theLaCzar•7/12/23, 3:13 PM

hmm serverless framework doesnt work much on any other cloud

T

theLaCzar•7/12/23, 3:14 PM

seems it heavily favors AWS

M

Matt Silverlock•7/12/23, 6:13 PM

FYI — if you're using D1's new
--experimental-backend
--experimental-backend
--experimental-backend
--experimental-backend and want to try D1's new Time Travel feature — you now can. This is a work-in-progress but sharing here. This preview branch may break at any time.

You can now use a temporary wrangler branch
```
npx wrangler@d1 d1 time-travel restore DB_NAME --timestamp=(unix timestamp|RFC3339 timestamp)
```
npx wrangler@d1 d1 time-travel restore DB_NAME --timestamp=(unix timestamp|RFC3339 timestamp)
```
npx wrangler@d1 d1 time-travel restore DB_NAME --timestamp=(unix timestamp|RFC3339 timestamp)
```
npx wrangler@d1 d1 time-travel restore DB_NAME --timestamp=(unix timestamp|RFC3339 timestamp)
to restore a database to any minute within the last 30 days.
Our preview documentation is at https://silverlock-d1-time-travel.cloudflare-docs-7ou.pages.dev/d1/learning/time-travel/#docs-content
For the initial release,
```
d1 time-travel restore DB_NAME
```
d1 time-travel restore DB_NAME
```
d1 time-travel restore DB_NAME
```
d1 time-travel restore DB_NAME
overwrites your existing database (with confirmation). In the future, we'll allow you to fork and/or clone to a new DB or over the top of an existing database.
You can read more about Time Travel from our original announcement post back in May: https://blog.cloudflare.com/d1-turning-it-up-to-11/#time-travel

LLiudmila is it correct that the plan is to make time travel default and thus we won't nee...

M

Matt Silverlock•7/12/23, 8:46 PM

Per the docs, yes: Time Travel will be the default. We address durability by writing databases to cold (object) storage: this is where idle databases are restored from.

Way too early to comment on anything non-JSON. Unlikely to be Protobuf.

H

Hello, I’m Allie!•7/12/23, 8:56 PM

https://capnproto.org/ time! /s

S

slowpoke•7/12/23, 10:20 PM

If I already created a D1 database on the dashboard, how can I interact with it during local development?

S

slowpoke•7/12/23, 10:22 PM

my wrangler.toml is this:

[[d1_databases]]
binding="DB"
database_name = "mydbname"
database_id = "db id string from the dashboard"

[[d1_databases]]
binding="DB"
database_name = "mydbname"
database_id = "db id string from the dashboard"

[[d1_databases]]
binding="DB"
database_name = "mydbname"
database_id = "db id string from the dashboard"

[[d1_databases]]
binding="DB"
database_name = "mydbname"
database_id = "db id string from the dashboard"

although when doing wrangler pages dev, it's undefined

S

slowpoke•7/12/23, 10:22 PM

honestly I'm just completely lost

S

slowpoke•7/13/23, 12:06 AM

Ok I've figured out what I need to do now, but it does feel janky.

S

slowpoke•7/13/23, 12:08 AM

I have my binding working in Qwik, once the site is uploaded to Cloudflare Pages.

However, locally, the only way I can see testing with the "same" D1 db that's on cloudflare is by making a

bootstrap.sql

bootstrap.sql

bootstrap.sql

bootstrap.sql /

db-setup.sql

db-setup.sql

db-setup.sql

db-setup.sql similar to the one in the tutorial, executing that with fake data, and going on from there.

Sslowpoke If I already created a D1 database on the dashboard, how can I interact with it ...

M

Matt Silverlock•7/13/23, 2:05 AM

Something the Pages team is working on improving. In the meantime (and this is not officially supported; could break/change in the future):

Create a
```
wrangler.toml
```
```
wrangler.toml
```
with a
```
preview_database_id
```
preview_database_id
```
preview_database_id
```
preview_database_id
key that maps to the same name used for
```
--d1=<BINDING>
```
--d1=<BINDING>
```
--d1=<BINDING>
```
--d1=<BINDING>
you pass to
```
wrangler pages dev
```
wrangler pages dev
```
wrangler pages dev
```
wrangler pages dev
- e.g. "DB" or "PROD_DB" (as appropriate)
Execute
```
wrangler d1
```
wrangler d1
```
wrangler d1
```
wrangler d1
commands with the
```
--local
```
--local
```
--local
```
--local
flag against that local database to seed it / run migrations / etc - e.g.
```
wrangler d1 execute db-name --file=seed.sql --local
```
wrangler d1 execute db-name --file=seed.sql --local
```
wrangler d1 execute db-name --file=seed.sql --local
```
wrangler d1 execute db-name --file=seed.sql --local
Ensure that you pass
```
--d1=BINDING_NAME
```
--d1=BINDING_NAME
```
--d1=BINDING_NAME
```
--d1=BINDING_NAME
to any
```
wrangler pages dev
```
wrangler pages dev
```
wrangler pages dev
```
wrangler pages dev
commands so that Pages references the same local DB.

Sslowpoke I have my binding working in Qwik, once the site is uploaded to Cloudflare Pages...

M

Matt Silverlock•7/13/23, 2:06 AM

Your

wrangler.toml

wrangler.toml

doesn't need anything more than just a

[[d1_database]]

[[d1_database]]

[[d1_database]]

[[d1_database]] definition: e.g. https://github.com/elithrar/remix-plus-d1/blob/main/wrangler.toml#L1-L5

K

katopz•7/13/23, 4:21 AM

My feedback

# This will yell Unknown argument: local
wrangler d1 create SUBSCRIPTIONS --local

# That's OK, it's local by default so I try
wrangler d1 create SUBSCRIPTIONS

# Then migrate...
wrangler d1 migrations apply SUBSCRIPTIONS

# Then run the app and got...
Uncaught (async) Error: D1_ERROR: Error: SqliteError: no such table: Subscriptions

# So i realized that I will need --local for migration (Took me an hour)
wrangler d1 migrations apply SUBSCRIPTIONS --local

# This will yell Unknown argument: local
wrangler d1 create SUBSCRIPTIONS --local

# That's OK, it's local by default so I try
wrangler d1 create SUBSCRIPTIONS

# Then migrate...
wrangler d1 migrations apply SUBSCRIPTIONS

# Then run the app and got...
Uncaught (async) Error: D1_ERROR: Error: SqliteError: no such table: Subscriptions

# So i realized that I will need --local for migration (Took me an hour)
wrangler d1 migrations apply SUBSCRIPTIONS --local

# This will yell Unknown argument: local
wrangler d1 create SUBSCRIPTIONS --local

# That's OK, it's local by default so I try
wrangler d1 create SUBSCRIPTIONS

# Then migrate...
wrangler d1 migrations apply SUBSCRIPTIONS

# Then run the app and got...
Uncaught (async) Error: D1_ERROR: Error: SqliteError: no such table: Subscriptions

# So i realized that I will need --local for migration (Took me an hour)
wrangler d1 migrations apply SUBSCRIPTIONS --local

# This will yell Unknown argument: local
wrangler d1 create SUBSCRIPTIONS --local

# That's OK, it's local by default so I try
wrangler d1 create SUBSCRIPTIONS

# Then migrate...
wrangler d1 migrations apply SUBSCRIPTIONS

# Then run the app and got...
Uncaught (async) Error: D1_ERROR: Error: SqliteError: no such table: Subscriptions

# So i realized that I will need --local for migration (Took me an hour)
wrangler d1 migrations apply SUBSCRIPTIONS --local

So some is needed

--local

--local

--local

--local some is not. My expectation is all should default to

--local

--local

--local

--local and

--remote

--remote

--remote

--remote when needed.

# local
wrangler d1 create SUBSCRIPTIONS
wrangler d1 migrations apply SUBSCRIPTIONS

# remote
wrangler d1 create SUBSCRIPTIONS --remote
wrangler d1 migrations apply SUBSCRIPTIONS --remote

# local
wrangler d1 create SUBSCRIPTIONS
wrangler d1 migrations apply SUBSCRIPTIONS

# remote
wrangler d1 create SUBSCRIPTIONS --remote
wrangler d1 migrations apply SUBSCRIPTIONS --remote

# local
wrangler d1 create SUBSCRIPTIONS
wrangler d1 migrations apply SUBSCRIPTIONS

# remote
wrangler d1 create SUBSCRIPTIONS --remote
wrangler d1 migrations apply SUBSCRIPTIONS --remote

# local
wrangler d1 create SUBSCRIPTIONS
wrangler d1 migrations apply SUBSCRIPTIONS

# remote
wrangler d1 create SUBSCRIPTIONS --remote
wrangler d1 migrations apply SUBSCRIPTIONS --remote

Would be nicer.

Kkatopz My feedback ``` # This will yell Unknown argument: local wrangler d1 create SUBS...

M

Matt Silverlock•7/13/23, 10:40 AM

Remote is always the default for any non-local-dev (wrangler dev, wrangler pages dev) commands. You’re creating databases: you don’t want to think you’ve created one and find out on deploy it doesn’t exist.

S

samratarmas•7/13/23, 11:37 AM

Hi team, does D1 support using sqlite extensions? For example, if I want to use full text search capability there is a built in Fts5 extension: https://www.sqlite.org/fts5.html, can I use this in D1 without additional configuration?

S

samratarmas•7/13/23, 11:47 AM

And second question, is js transactions coming anytime this year or is that a much longer term feature? ( read a value from d1, use that read value to do some calculation then write the result back, all inside a transaction)

MMatt Silverlock Your `wrangler.toml` doesn't need anything more than just a `[[d1_database]]` de...

S

slowpoke•7/13/23, 1:04 PM

Thanks for your response :) Yep! This is basically what I have right now and it works great!

I suppose once DB backups are available to download, it would be nice to download that to use for local development. Making a seed in the meantime is nice anyways because it declares the database structure for reference

Ssamratarmas Hi team, does D1 support using sqlite extensions? For example, if I want to use ...

M

Matt Silverlock•7/13/23, 1:08 PM

FTS5 is built in. On our list to include in our documentation, but works exactly as the SQLite docs say. We also support JSON5 - https://developers.cloudflare.com/d1/learning/querying-json/

Query JSON · Cloudflare D1 docs

D1 has built-in support for querying and parsing JSON data stored within a database. This enables you to:

M

micromashor•7/13/23, 6:06 PM

I keep randomly encountering this error while using

wrangler d1 execute

wrangler d1 execute

wrangler d1 execute

wrangler d1 execute: (some details redacted)

✘ [ERROR] A request to the Cloudflare API (/accounts/<ACCOUNT_ID>/d1/database/<DATABASE_ID>/query) failed.

  Internal error: Error sending request to D1 for database <DATABASE_ID>
  [code: 7501]

✘ [ERROR] A request to the Cloudflare API (/accounts/<ACCOUNT_ID>/d1/database/<DATABASE_ID>/query) failed.

  Internal error: Error sending request to D1 for database <DATABASE_ID>
  [code: 7501]

✘ [ERROR] A request to the Cloudflare API (/accounts/<ACCOUNT_ID>/d1/database/<DATABASE_ID>/query) failed.

  Internal error: Error sending request to D1 for database <DATABASE_ID>
  [code: 7501]

✘ [ERROR] A request to the Cloudflare API (/accounts/<ACCOUNT_ID>/d1/database/<DATABASE_ID>/query) failed.

  Internal error: Error sending request to D1 for database <DATABASE_ID>
  [code: 7501]

M

micromashor•7/13/23, 6:09 PM

it's also weird bc about half the time I see the error, the query went through successfully

M

Matt Silverlock•7/13/23, 6:55 PM

Are you using the new experimental backend? You can (should) log

error.cause.message

error.cause.message

error.cause.message

error.cause.message as well - https://developers.cloudflare.com/d1/platform/client-api/#errors

D1 client API · Cloudflare D1 docs

As part of our Client API, both static and prepared statements are supported. Best practice is to use prepared statements which are precompiled …

Original message was deleted

M

micromashor•7/14/23, 1:27 AM

yes that's correct - it was produced by wrangler

MMatt Silverlock Are you using the new experimental backend? You can (should) log `error.cause.me...

M

micromashor•7/14/23, 2:14 AM

and yes, I am using the experimental backend. I was trying to do an

INSERT

INSERT

operation with fewer than 20 rows, so I can't imagine it was failing due to any kind of size limit

Mmicromashor and yes, I am using the experimental backend. I was trying to do an `INSERT` ope...

M

Matt Silverlock•7/14/23, 11:35 AM

Without more details it’s hard to debug. Is the data static? Not all rows are created equal.

I just noticed a table called `_cf_KV` on every D1 database. This table has two columns, `key` and `

Similar Threads