Update: Solved by adding `--remote`. See discession below. I'm not sure I understand how to do

Update: Solved by adding

--remote

--remote

. See discession below.

I'm not sure I understand how to do a migration.

Scenario:

I've never used a migration before. I've modified a few tables in my local db, done my development code to deal with these changes.
Now, I've pushed my code and it's active in the preview environment of CF pages.

Migration Attempt:

I created a migration file using

wrangler d1 migrations create preview-say-my-name move-authentication-to-workos

wrangler d1 migrations create preview-say-my-name move-authentication-to-workos

wrangler d1 migrations create preview-say-my-name move-authentication-to-workos

wrangler d1 migrations create preview-say-my-name move-authentication-to-workos.

I edited the migration file with:

ALTER TABLE customers ADD COLUMN auth_service_user_id TEXT;
DROP TABLE IF EXISTS `magiclinks`;
DROP TABLE IF EXISTS `sessions`;
DROP TABLE IF EXISTS `login_passcodes`;

ALTER TABLE customers ADD COLUMN auth_service_user_id TEXT;
DROP TABLE IF EXISTS `magiclinks`;
DROP TABLE IF EXISTS `sessions`;
DROP TABLE IF EXISTS `login_passcodes`;

ALTER TABLE customers ADD COLUMN auth_service_user_id TEXT;
DROP TABLE IF EXISTS `magiclinks`;
DROP TABLE IF EXISTS `sessions`;
DROP TABLE IF EXISTS `login_passcodes`;

ALTER TABLE customers ADD COLUMN auth_service_user_id TEXT;
DROP TABLE IF EXISTS `magiclinks`;
DROP TABLE IF EXISTS `sessions`;
DROP TABLE IF EXISTS `login_passcodes`;

When I run

wrangler d1 migrations apply preview-say-my-name

wrangler d1 migrations apply preview-say-my-name

wrangler d1 migrations apply preview-say-my-name

wrangler d1 migrations apply preview-say-my-name, i get:

Migration 0001_move-authentication-to-workos.sql failed with the following errors:
✘ [ERROR] no such table: customers

Questions:

It seems that my original database and all its data is not used in the migration. So, it seems my migrations file is required to contain all the original schema and inserts for the exiting data.

Is that correct? This sure makes life painful and sort of makes migrations useless.

Do I have to continue doing this for every migration or once I've done the first, will the system be smart enough to carry data forward?

Feedback:

There's just not enough explanation of this in the docs.

JJustinNoel **Update:** Solved by adding `--remote`. See discession below. I'm not sure I ...

Hello, I’m Allie!•6/29/24, 2:32 PM

Just to confirm, is this local mode?

HHello, I’m Allie!Just to confirm, is this local mode?

JustinNoelOP•6/29/24, 2:42 PM

No. preview/staging is what i'm attempting right now.

Original message was deleted

JustinNoelOP•6/29/24, 2:47 PM

That did it! i was assuming it would have read that info from my

wrangler.toml

wrangler.toml

because it had to add it for

wrangler d1 migrations create preview-say-my-name move-authentication-to-workos

wrangler d1 migrations create preview-say-my-name move-authentication-to-workos

.

Thanks so much!

Aakira.eth const env = getMiniflareBindings() anyone know why this isn't returning my db bi...

Matt Silverlock•6/29/24, 3:41 PM

What version are you on? https://developers.cloudflare.com/d1/build-with-d1/local-development/#miniflarehttps://developers.cloudflare.com/d1/build-with-d1/local-development/#miniflare

Matt Silverlock•6/29/24, 3:42 PM

(Can’t make clickable as Discord interprets the anchor as a channel link)

Matt Silverlock•6/29/24, 3:42 PM

https://developers.cloudflare.com/d1/build-with-d1/local-development/ - scroll down

Cloudflare Docs

Local development · Cloudflare D1 docs

D1 has fully-featured support for local development, running the same version of D1 as Cloudflare runs globally. Local development uses Wrangler, the …

FFriday Candour Maybe you are solving a problem wrongly.

Tilon•6/30/24, 12:47 PM

Maybe you are in the wrong forum...

Ssurgery18 Ok, that is what I was thinking. Gonna dread doing the conversion since it is AL...

Jason Hostetter•6/30/24, 4:04 PM

Before you undertake a big conversion, I’d recommend looking closely at D1’s limitations in the docs, to make sure the data you’re migrating will fall within those limits. For example, there is a row size limit of 1MB, which is pretty big, but if you are doing things like saving unbounded data from user input, or like base64 encoded images you could hit that.

JJason Hostetter Before you undertake a big conversion, I’d recommend looking closely at D1’s lim...

surgery18•6/30/24, 6:49 PM

Yeah, I should be ok with that limit. Any image stuff just gets saved into a R2 bucket and a key gets saved in the DB. So I should be ok with the limit.

malone•6/30/24, 6:50 PM

what are people doing to unit test with D1 databases?

I ideally want to just pass a mock d1 database to my Hono app i import, and not deal with cloudflare wrangler/miniflare as much as possible. i want to do this so i can use Bun's tests which are really fast.

is this possible right now? if I use miniflare@3 it hangs when i request a DB, and if I use miniflare@2 it looks like i need to run the whole worker and it wants me to use a vitest or jest integration

malone•6/30/24, 6:52 PM

i want to do some large scale changes to my side project and didn't write any tests, so i'm trying to get some tests in before making big changes

malone•6/30/24, 9:04 PM

I ended up just making my own D1Database conforming thing that wraps bun:sqlite, good enough for my use

Oli•6/30/24, 10:02 PM

Hi! Just joined here, is the "raw" endpoint for the D1 rest api broken?

Always getting:
" "errors": [
{
"code": 7400,
"message": "The request is malformed: failed to decode id"
}
"

And i saw something on it not working on github, but the issues were not super recent. It dosent work when using wrangler neither..

Disciple of Caoximeihao•7/1/24, 9:29 AM

how to use cloudflare worker localhost and d1 remote?

Disciple of Caoximeihao•7/1/24, 9:29 AM

help

DDisciple of Caoximeihao how to use cloudflare worker localhost and d1 remote?

Hello, I’m Allie!•7/1/24, 9:30 AM

You can use the —remote flag, which will run the Worker on Cloudflare, but tunnel it back down to localhost, if that is what you mean?

HHello, I’m Allie!You can use the —remote flag, which will run the Worker on Cloudflare, but tunne...

Disciple of Caoximeihao•7/1/24, 9:43 AM

amazing, thanks you

OOli Hi! Just joined here, is the "raw" endpoint for the D1 rest api broken? Always ...

Matt Silverlock•7/1/24, 2:02 PM

Can you share your request? The older GitHub issues were from before the raw endpoint existed

Iibo cloudflare d1 sqlite db good for e commerce production app? Is there a problem w...

malone•7/1/24, 3:55 PM

go for it

Allan Cassio•7/1/24, 5:21 PM

Allan Cassio•7/1/24, 5:22 PM

my function is called twice.

Allan Cassio•7/1/24, 5:22 PM

samebody know why this?

TrueBrain•7/1/24, 5:26 PM

once for

, once for /favicon.ico/favicon.ico?

TTrueBrain once for `/`, once for `/favicon.ico`? 🙂

Allan Cassio•7/1/24, 5:29 PM

/favicon.ico

/favicon.ico

called, my route is

Allan Cassio•7/1/24, 5:56 PM

????

Allan Cassio•7/1/24, 5:58 PM

local is ok. but when I use deploy return this error

Gerbuuun•7/1/24, 10:05 PM

That are a lot of rows

MMatt Silverlock https://developers.cloudflare.com/d1/build-with-d1/local-development/ - scroll d...

Chinoman10•7/1/24, 10:17 PM

So... how can I

--remote

--remote

with

wrangler pages dev

wrangler pages dev

pages

pages

doesn't seem to have that option

and I'd like to access my D1 while testing locally with my Pages project.

Bbun `wrangler d1 export YOUR_DB_NAME --output db.sql --remote && wrangler d1 execute...

Chinoman10•7/2/24, 9:17 AM

This has "saved" me

thank you so much for this

I'll add this to my package.json so I don't forget!

Flare•7/2/24, 9:29 PM

Please do not post your question in multiple channels per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

MMatt Silverlock Can you share your request? The older GitHub issues were from _before_ the raw e...

Oli•7/2/24, 10:53 PM

Ive tried using multiple types of request, i sat for like 2h..
All endpoitns for d1 works except query/raw
Also, note that the queries worked when running d1 locally.

You can see how the request is formed here(line 147):
https://github.com/hlpmenu/cloudflare-cli/blob/test/cf_api/D1.go#L147

Also added comments for other methods used. The isValidSql() function creates a sqlite db file and tries the query against it beforehand, which has 0 issues.
The db id, and account id is used for all requests, which means its not possible theres where the issue lies

Tried:
Using curl and go:
- Auth using api key and api token + created a new token.

- Sending the demo requests from the rest docs,

- Created a new d1 db to make sure it wasent a temporary bug.

- Tried to construct the json request manually in go, reading from a file, logged the query and request, used a json string, unescaped it, escaped it, different quote types, trimmed any potential whitespace.

- For id, which it complains about, removed the "-"

Curl:(copied from terminal now, so the format is a bit messed up, but you get the gest of it)
curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/d1/database/$DB_ID/raw" \ -H "X-Auth-Key: $CLOUDFLARE_API_KEY" \
-H "X-Auth-Email: $CLOUDFLARE_ACCOUNT_EMAIL" \
-H "Content-Type: application/json" \
--data '{"sql":"SELECT email FROM users;"}'

Wrangler with --command:
~/D/gem1 (production)> wrangler d1 execute $DB_NAME --command "SELECT * FROM *;" --remote

wrangler 3.62.0
-------------------

✘ [ERROR] Couldn't find DB with name '[REDACTED]'

Wrangler with file:
Dont have the output but it was a generic error message.

GitHub

cloudflare-cli/cf_api/D1.go at test · hlpmenu/cloudflare-cli

An interactive CLI for managing Cloudflare wokers, pages, D1, R2 and more! - hlpmenu/cloudflare-cli

OOli Ive tried using multiple types of request, i sat for like 2h.. All endpoitns fo...

Matt Silverlock•7/2/24, 11:02 PM

If you are using

—remote

—remote

but get “couldn’t find DB” - that’s because you didn’t create it. What does wrangler d1 listwrangler d1 list show?

By default,

execute

execute

operates on local databases first.

OOli Ive tried using multiple types of request, i sat for like 2h.. All endpoitns fo...

Matt Silverlock•7/2/24, 11:03 PM

I also don’t understand if you are saying the curl works or not. Does it work?

MMatt Silverlock I also don’t understand if you are saying the curl works or not. Does it work?

Oli•7/2/24, 11:07 PM

Sorry, i was a bit unclear, curl does not work..
What i meant was that every endpoint works, except query and raw.

wrangler d1 list shows all listed db:s.
I got the id from the dashboard to make sure wrangler was not outputting any local uuids.

The db id, db name ia all set by env vars.

OOli Sorry, i was a bit unclear, curl does not work.. What i meant was that every en...

Matt Silverlock•7/2/24, 11:12 PM

Set them directly, just to remove any env var expansion mistakes.

OOli Sorry, i was a bit unclear, curl does not work.. What i meant was that every en...

Matt Silverlock•7/2/24, 11:13 PM

It’s also unclear why curl doesn’t work. What error?

Matt Silverlock•7/2/24, 11:13 PM

My suggestion:

1. Get the basic curl working first.
2. Then you can move to env vars
3. Then your Go app.

Oli•7/2/24, 11:23 PM

I think you misunderstood, no method works, the error is the api response:
"message": "The request is malformed: failed to decode id"

Nope, tried it multiple times now, including removing any linebreaks, same issue.

The response is:
{
"result": null,
"success": false,
"errors": [
{
"code": 7400,
"message": "The request is malformed: failed to decode id"
}
],
"messages": []
}

Oli•7/3/24, 12:11 AM

Ok, so i managed to get it to work, but not with that db..?

Its super inconstent..

il sanitize some logs and upload

Oli•7/3/24, 12:38 AM

Requests:
https://gist.github.com/hlpmenu/fc0a95f514a824a6b4e937a2169f306a

Tried with a new db too.
Same thing, works for those, not for that one.
Obviosly doublechecked id with both wrangler and dashboard

Gist

req.log

GitHub Gist: instantly share code, notes, and snippets.

OOli Requests: https://gist.github.com/hlpmenu/fc0a95f514a824a6b4e937a2169f306a Trie...

Matt Silverlock•7/3/24, 4:24 AM

Without the contents of the

cf

cf

script or the database ID there’s not much we can do to help. Are you running an old alpha database at all? (see the docs for instructions on how to discern)

I suggest opening an issue on the workers-sdk repo with the database ID (which is safe to post publicly), account ID, and your script exactly as you are running it.

Otherwise - flying blind.

temporaryatearth•7/3/24, 8:43 AM

is the type TIMESTAMP really a timestamp without timezone? I would have to use this to check records several minutes later after they are created

temporaryatearth•7/3/24, 8:43 AM

since I can't find documents on what data types that D1 supports.

Ttemporaryatearth is the type TIMESTAMP really a timestamp without timezone? I would have to use t...

Max (@rozenmd)•7/3/24, 8:43 AM

TIMESTAMP isn't an sqlite type, i'm guessing this is a library on top of D1?

Update: Solved by adding `--remote`. See discession below. I'm not sure I understand how to do

Similar Threads

Similar Threads

Similar Threads

**Update:** Solved by adding `--remote`. See discession below. I'm not sure I understand how to do

Similar Threads

Similar Threads

Similar Threads

Update: Solved by adding `--remote`. See discession below. I'm not sure I understand how to do