Before you undertake a big conversion, I’d recommend looking closely at D1’s limitations in the docs

Before you undertake a big conversion, I’d recommend looking closely at D1’s limitations in the docs, to make sure the data you’re migrating will fall within those limits. For example, there is a row size limit of 1MB, which is pretty big, but if you are doing things like saving unbounded data from user input, or like base64 encoded images you could hit that.

JJason Hostetter Before you undertake a big conversion, I’d recommend looking closely at D1’s lim...

surgery18•6/30/24, 6:49 PM

Yeah, I should be ok with that limit. Any image stuff just gets saved into a R2 bucket and a key gets saved in the DB. So I should be ok with the limit.

malone•6/30/24, 6:50 PM

what are people doing to unit test with D1 databases?

I ideally want to just pass a mock d1 database to my Hono app i import, and not deal with cloudflare wrangler/miniflare as much as possible. i want to do this so i can use Bun's tests which are really fast.

is this possible right now? if I use miniflare@3 it hangs when i request a DB, and if I use miniflare@2 it looks like i need to run the whole worker and it wants me to use a vitest or jest integration

malone•6/30/24, 6:52 PM

i want to do some large scale changes to my side project and didn't write any tests, so i'm trying to get some tests in before making big changes

malone•6/30/24, 9:04 PM

I ended up just making my own D1Database conforming thing that wraps bun:sqlite, good enough for my use

Oli•6/30/24, 10:02 PM

Hi! Just joined here, is the "raw" endpoint for the D1 rest api broken?

Always getting:
" "errors": [
{
"code": 7400,
"message": "The request is malformed: failed to decode id"
}
"

And i saw something on it not working on github, but the issues were not super recent. It dosent work when using wrangler neither..

Disciple of Caoximeihao•7/1/24, 9:29 AM

how to use cloudflare worker localhost and d1 remote?

Disciple of Caoximeihao•7/1/24, 9:29 AM

help

DDisciple of Caoximeihao how to use cloudflare worker localhost and d1 remote?

Hello, I’m Allie!•7/1/24, 9:30 AM

You can use the —remote flag, which will run the Worker on Cloudflare, but tunnel it back down to localhost, if that is what you mean?

HHello, I’m Allie!You can use the —remote flag, which will run the Worker on Cloudflare, but tunne...

Disciple of Caoximeihao•7/1/24, 9:43 AM

amazing, thanks you

OOli Hi! Just joined here, is the "raw" endpoint for the D1 rest api broken? Always ...

Matt Silverlock•7/1/24, 2:02 PM

Can you share your request? The older GitHub issues were from before the raw endpoint existed

Iibo cloudflare d1 sqlite db good for e commerce production app? Is there a problem w...

malone•7/1/24, 3:55 PM

go for it

Allan Cassio•7/1/24, 5:21 PM

Allan Cassio•7/1/24, 5:22 PM

my function is called twice.

Allan Cassio•7/1/24, 5:22 PM

samebody know why this?

TrueBrain•7/1/24, 5:26 PM

once for

, once for /favicon.ico/favicon.ico?

TTrueBrain once for `/`, once for `/favicon.ico`? 🙂

Allan Cassio•7/1/24, 5:29 PM

/favicon.ico

/favicon.ico

called, my route is

Allan Cassio•7/1/24, 5:56 PM

????

Allan Cassio•7/1/24, 5:58 PM

local is ok. but when I use deploy return this error

Gerbuuun•7/1/24, 10:05 PM

That are a lot of rows

MMatt Silverlock https://developers.cloudflare.com/d1/build-with-d1/local-development/ - scroll d...

Chinoman10•7/1/24, 10:17 PM

So... how can I

--remote

--remote

with

wrangler pages dev

wrangler pages dev

pages

pages

doesn't seem to have that option

and I'd like to access my D1 while testing locally with my Pages project.

Bbun `wrangler d1 export YOUR_DB_NAME --output db.sql --remote && wrangler d1 execute...

Chinoman10•7/2/24, 9:17 AM

This has "saved" me

thank you so much for this

I'll add this to my package.json so I don't forget!

Flare•7/2/24, 9:29 PM

Please do not post your question in multiple channels per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

MMatt Silverlock Can you share your request? The older GitHub issues were from _before_ the raw e...

Oli•7/2/24, 10:53 PM

Ive tried using multiple types of request, i sat for like 2h..
All endpoitns for d1 works except query/raw
Also, note that the queries worked when running d1 locally.

You can see how the request is formed here(line 147):
https://github.com/hlpmenu/cloudflare-cli/blob/test/cf_api/D1.go#L147

Also added comments for other methods used. The isValidSql() function creates a sqlite db file and tries the query against it beforehand, which has 0 issues.
The db id, and account id is used for all requests, which means its not possible theres where the issue lies

Tried:
Using curl and go:
- Auth using api key and api token + created a new token.

- Sending the demo requests from the rest docs,

- Created a new d1 db to make sure it wasent a temporary bug.

- Tried to construct the json request manually in go, reading from a file, logged the query and request, used a json string, unescaped it, escaped it, different quote types, trimmed any potential whitespace.

- For id, which it complains about, removed the "-"

Curl:(copied from terminal now, so the format is a bit messed up, but you get the gest of it)
curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/d1/database/$DB_ID/raw" \ -H "X-Auth-Key: $CLOUDFLARE_API_KEY" \
-H "X-Auth-Email: $CLOUDFLARE_ACCOUNT_EMAIL" \
-H "Content-Type: application/json" \
--data '{"sql":"SELECT email FROM users;"}'

Wrangler with --command:
~/D/gem1 (production)> wrangler d1 execute $DB_NAME --command "SELECT * FROM *;" --remote

wrangler 3.62.0
-------------------

✘ [ERROR] Couldn't find DB with name '[REDACTED]'

Wrangler with file:
Dont have the output but it was a generic error message.

GitHub

cloudflare-cli/cf_api/D1.go at test · hlpmenu/cloudflare-cli

An interactive CLI for managing Cloudflare wokers, pages, D1, R2 and more! - hlpmenu/cloudflare-cli

OOli Ive tried using multiple types of request, i sat for like 2h.. All endpoitns fo...

Matt Silverlock•7/2/24, 11:02 PM

If you are using —remote—remote but get “couldn’t find DB” - that’s because you didn’t create it. What does wrangler d1 listwrangler d1 list show?

By default,

execute

execute

operates on local databases first.

OOli Ive tried using multiple types of request, i sat for like 2h.. All endpoitns fo...

Matt Silverlock•7/2/24, 11:03 PM

I also don’t understand if you are saying the curl works or not. Does it work?

MMatt Silverlock I also don’t understand if you are saying the curl works or not. Does it work?

Oli•7/2/24, 11:07 PM

Sorry, i was a bit unclear, curl does not work..
What i meant was that every endpoint works, except query and raw.

wrangler d1 list shows all listed db:s.
I got the id from the dashboard to make sure wrangler was not outputting any local uuids.

The db id, db name ia all set by env vars.

OOli Sorry, i was a bit unclear, curl does not work.. What i meant was that every en...

Matt Silverlock•7/2/24, 11:12 PM

Set them directly, just to remove any env var expansion mistakes.

OOli Sorry, i was a bit unclear, curl does not work.. What i meant was that every en...

Matt Silverlock•7/2/24, 11:13 PM

It’s also unclear why curl doesn’t work. What error?

Matt Silverlock•7/2/24, 11:13 PM

My suggestion:

1. Get the basic curl working first.
2. Then you can move to env vars
3. Then your Go app.

Oli•7/2/24, 11:23 PM

I think you misunderstood, no method works, the error is the api response:
"message": "The request is malformed: failed to decode id"

Nope, tried it multiple times now, including removing any linebreaks, same issue.

The response is:
{
"result": null,
"success": false,
"errors": [
{
"code": 7400,
"message": "The request is malformed: failed to decode id"
}
],
"messages": []
}

Oli•7/3/24, 12:11 AM

Ok, so i managed to get it to work, but not with that db..?

Its super inconstent..

il sanitize some logs and upload

Oli•7/3/24, 12:38 AM

Requests:
https://gist.github.com/hlpmenu/fc0a95f514a824a6b4e937a2169f306a

Tried with a new db too.
Same thing, works for those, not for that one.
Obviosly doublechecked id with both wrangler and dashboard

Gist

req.log

GitHub Gist: instantly share code, notes, and snippets.

OOli Requests: https://gist.github.com/hlpmenu/fc0a95f514a824a6b4e937a2169f306a Trie...

Matt Silverlock•7/3/24, 4:24 AM

Without the contents of the cfcf script or the database ID there’s not much we can do to help. Are you running an old alpha database at all? (see the docs for instructions on how to discern)

I suggest opening an issue on the workers-sdk repo with the database ID (which is safe to post publicly), account ID, and your script exactly as you are running it.

Otherwise - flying blind.

temporaryatearth•7/3/24, 8:43 AM

is the type TIMESTAMP really a timestamp without timezone? I would have to use this to check records several minutes later after they are created

temporaryatearth•7/3/24, 8:43 AM

since I can't find documents on what data types that D1 supports.

Ttemporaryatearth is the type TIMESTAMP really a timestamp without timezone? I would have to use t...

Max (@rozenmd)•7/3/24, 8:43 AM

TIMESTAMP isn't an sqlite type, i'm guessing this is a library on top of D1?

Ttemporaryatearth since I can't find documents on what data types that D1 supports.

Max (@rozenmd)•7/3/24, 8:44 AM

https://www.sqlite.org/datatype3.html

temporaryatearth•7/3/24, 8:44 AM

I don't know, i created the table with a default current_timestampdefault current_timestamp and it is working well

Max (@rozenmd)•7/3/24, 8:45 AM

If the default value of a column is CURRENT_TIME, CURRENT_DATE or CURRENT_TIMESTAMP, then the value used in the new row is a text representation of the current UTC date and/or time. For CURRENT_TIME, the format of the value is "HH:MM:SS". For CURRENT_DATE, "YYYY-MM-DD". The format for CURRENT_TIMESTAMP is "YYYY-MM-DD HH:MM:SS".

from https://www.sqlite.org/lang_createtable.html

temporaryatearth•7/3/24, 8:47 AM

OK, I think it is better to use INTEGERINTEGER and maintain the value by code logic..

temporaryatearth•7/3/24, 8:47 AM

thank you @Max (@rozenmd)

maguayo•7/4/24, 7:37 AM

I have an SQLite on my VPS. Can I create an script or something to push my whole database (50MB) to Coudflare D1? That way having a similar version in the edge as kind of cache?

Mmaguayo I have an SQLite on my VPS. Can I create an script or something to push my whole...

Hard@Work•7/4/24, 9:12 AM

Maybe do an initial sync, then mirror all commands to the local SQLite db to D1?

WillTolmie•7/4/24, 10:17 AM

Strange one this - my worker still seems to have access to a deleted database. I was wondering why my API was working fine (new rows being added and being queried fine) but new rows weren't showing in the CF dashboard

WillTolmie•7/4/24, 10:17 AM

I tried D1 export from the root of the worker via wrangler and it complained the database didn't exist

Before you undertake a big conversion, I’d recommend looking closely at D1’s limitations in the docs

Similar Threads

Similar Threads

Similar Threads