anyone knows how to do full database backup with wrangler ?

YYokoWasis anyone knows how to do full database backup with wrangler ?

J

James•11/22/23, 5:12 AM

You can’t yet directly with wrangler, but you can use this great community solution: https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

F

federo•11/22/23, 11:39 AM

can anyone help me understand how is this viable if a database can only hold 2gb of data?

Z

zegevlier•11/22/23, 11:42 AM

(not a team member, just from my knowledge as an outsider)
First: The goal is still to increase this during the beta. Second: With D1, you can have a lot of seperate databases, all with a limit of 2gb. If your application can support sharding the DB like this, there isn't really a limit to the amount of data you can store. To me it seems perfect for a lot of SaaS applications, where each customer, or even each project, would have their own database.

F

federo•11/22/23, 11:44 AM

so if a client would then need to make a query, the query would have to first look at which database to look into and perhaps query the data across databases?

Ffedero so if a client would then need to make a query, the query would have to first lo...

M

Max (@rozenmd)•11/22/23, 11:46 AM

a certain issue manager that many folks dislike handles multi-tenancy that way, yeah - one db per customer.

you keep a highly available store mapping something (customerId, a subdomain, etc) to the database ID, then pass the databaseID to your client (we're working on making the d1-api support this nicely)

M

Max (@rozenmd)•11/22/23, 11:48 AM

migrations are also a pain atm with this model: https://github.com/cloudflare/workers-sdk/issues/4471

F

federo•11/22/23, 11:48 AM

but if a customers needs way more than 2gb of data, then do you see this viable?
I am trying to setup a flow where a js script on websites sends events data to a cloudflare worker, from which I am trying to understand how to best forward it to a database

M

Max (@rozenmd)•11/22/23, 11:49 AM

for events data d1 is going to have a bad time, you might want to look into https://developers.cloudflare.com/analytics/analytics-engine/

Ffedero but if a customers needs way more than 2gb of data, then do you see this viable?...

Z

zegevlier•11/22/23, 11:49 AM

also see: d1-database

F

federo•11/22/23, 11:52 AM

analytics engine is not feasible either with a retention period of only 3 months

Zzegevlier also see: https://discord.com/channels/595317990191398933/992060581832032316/117...

F

federo•11/22/23, 12:04 PM

it seems like Hyperdrive is just for reads not writes

GGustavo One idea would be to aggregate and move to somewhere else

F

federo•11/22/23, 12:05 PM

how do you move it to somewhere else? I am struggling to understand how to take payloads coming into workers and load them into a solid database

Ffedero it seems like Hyperdrive is just for reads not writes

K

kian•11/22/23, 12:06 PM

Writes will just passthrough the caching and still take advantage of the Cloudflare backbone.

GGustavo I mean write to Analytics Engine, then have a cron job run each month to aggrega...

R

Rob M.•11/22/23, 12:29 PM

But what about WAE sampling? Looks like we can lose data in some cases.

RRob M.But what about WAE sampling? Looks like we can lose data in some cases.

M

Max (@rozenmd)•11/22/23, 12:46 PM

if you need every single row, i would just use postgres

Ffedero it seems like Hyperdrive is just for reads not writes

M

Matt Silverlock•11/22/23, 1:28 PM

Hyperdrive doesn’t restrict you. Curiously: why do you think it’s just reads?

Nnick_jordan When running a query from the UI Console I'm getting the following error... ```...

M

Max (@rozenmd)•11/22/23, 3:07 PM

That looks like an alpha database - the current default backend for D1 performs around 10-11x faster for writes, 20x faster for reads: [https://blog.cloudflare.com/d1-turning-it-up-to-11/]

A

ajgeiss0702•11/22/23, 5:25 PM

in the dashboard, is there a plan to add rows read/written so I can have an idea on what my bill would be? I know I can see it for the past week by going to the per-db metrics, but I would like to be able to see the total for the month since that what i'll be billed for once billing gets enabled

Aajgeiss0702 in the dashboard, is there a plan to add rows read/written so I can have an idea...

M

Max (@rozenmd)•11/22/23, 5:31 PM

That's a good point - I'll add it to our backlog

MMax (@rozenmd)That's a good point - I'll add it to our backlog

A

ajgeiss0702•11/22/23, 5:38 PM

ty!

Y

YokoWasisOP•11/23/23, 12:05 AM

Let's hope today D1 won't be down like yesterday. I made 4 replica of the database. It would be absurd of all of them going down.

Y

YokoWasisOP•11/23/23, 12:52 AM

My D1 database said too many queue operation. Any recommendation ? is D1 is not feasible for many users ?

T

Tesbarmix•11/23/23, 12:53 AM

Hello

Y

YokoWasisOP•11/23/23, 12:59 AM

What about kv ? does kv can handle R/W requests better than d1 ?

YYokoWasis My D1 database said too many queue operation. Any recommendation ? is D1 is not ...

M

Max (@rozenmd)•11/23/23, 8:54 AM

what's your use case?

MMax (@rozenmd)what's your use case?

Y

YokoWasisOP•11/23/23, 9:17 AM

student's national exam.

YYokoWasis student's national exam.

M

Max (@rozenmd)•11/23/23, 9:20 AM

as in, are you using D1 as a single database for an API that your frontend talks to? how many requests per second?

MMax (@rozenmd)as in, are you using D1 as a single database for an API that your frontend talks...

Y

YokoWasisOP•11/23/23, 9:54 AM

Yes, worker as a backend api. About 1k req per second.

H

Hearse•11/23/23, 11:17 AM

wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Could not read file: ./drizzle/0000_short_lockheed.sql

  ENOENT: no such file or directory, open
  '/Users/administrator/backend/project/drizzle/0000_short_lockheed.sql'
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Could not read file: ./drizzle/0000_short_lockheed.sql

  ENOENT: no such file or directory, open
  '/Users/administrator/backend/project/drizzle/0000_short_lockheed.sql'
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Could not read file: ./drizzle/0000_short_lockheed.sql

  ENOENT: no such file or directory, open
  '/Users/administrator/backend/project/drizzle/0000_short_lockheed.sql'
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Could not read file: ./drizzle/0000_short_lockheed.sql

  ENOENT: no such file or directory, open
  '/Users/administrator/backend/project/drizzle/0000_short_lockheed.sql'
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

H

Hearse•11/23/23, 11:18 AM

Why is it doing this when following https://github.com/drizzle-team/drizzle-orm/blob/main/examples/cloudflare-d1/README.md

GitHub

drizzle-orm/examples/cloudflare-d1/README.md at main · drizzle-team...

TypeScript ORM that feels like writing SQL. Contribute to drizzle-team/drizzle-orm development by creating an account on GitHub.

R

Rob M.•11/23/23, 11:19 AM

Is the file really there?

RRob M.Is the file really there?

H

Hearse•11/23/23, 11:20 AM

it doesnt need to be i thought .

RRob M.Is the file really there?

H

Hearse•11/23/23, 11:20 AM

https://orm.drizzle.team/docs/quick-sqlite/d1

Cloudflare D1 - Drizzle ORM

Drizzle ORM is a lightweight and performant TypeScript ORM with developer experience in mind.

H

Hearse•11/23/23, 11:21 AM

i thought the command creates it

H

Hearse•11/23/23, 11:22 AM

literally just run the command in the directory

H

Hearse•11/23/23, 11:22 AM

i have d1 setup in my wrangler config and its working fine, running this command in the same project folder doesnt work

H

Hearse•11/23/23, 11:23 AM

if i do create a dummy file, it literally say this.

mkdir drizzle
touch drizzle/0000_short_lockheed.sql
wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Error: must provide --command or --file.


If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
🪵  Logs were written to "/Users/administrator/Library/Preferences/.wrangler/logs/wrangler-2023-11-23_11-23-02_962.log"

mkdir drizzle
touch drizzle/0000_short_lockheed.sql
wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Error: must provide --command or --file.


If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
🪵  Logs were written to "/Users/administrator/Library/Preferences/.wrangler/logs/wrangler-2023-11-23_11-23-02_962.log"

mkdir drizzle
touch drizzle/0000_short_lockheed.sql
wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Error: must provide --command or --file.


If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
🪵  Logs were written to "/Users/administrator/Library/Preferences/.wrangler/logs/wrangler-2023-11-23_11-23-02_962.log"

mkdir drizzle
touch drizzle/0000_short_lockheed.sql
wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Error: must provide --command or --file.


If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
🪵  Logs were written to "/Users/administrator/Library/Preferences/.wrangler/logs/wrangler-2023-11-23_11-23-02_962.log"

H

Hearse•11/23/23, 11:24 AM

wrangler 3.17.1

wrangler 3.17.1

wrangler 3.17.1

wrangler 3.17.1

btw, it should be latest.

R

Rob M.•11/23/23, 11:44 AM

maybe, there is a newline symbol after "--"?

HHearse if i do create a dummy file, it literally say this. ``` mkdir drizzle touch driz...

M

Max (@rozenmd)•11/23/23, 12:06 PM

thats weird, do you mind filing a github issue? looks like a bug

M

Max (@rozenmd)•11/23/23, 12:07 PM

ohhhh

H

Hearse•11/23/23, 12:07 PM

Will do when i wake up

M

Max (@rozenmd)•11/23/23, 12:07 PM

no thats just a bad error message

M

Max (@rozenmd)•11/23/23, 12:07 PM

it throws "Error: must provide --command or --file" if there's an empty string too

M

Max (@rozenmd)•11/23/23, 12:08 PM

I think you need to run

npm run generate

npm run generate

npm run generate

npm run generate for drizzle to create the SQL

MMax (@rozenmd)I think you need to run `npm run generate` for drizzle to create the SQL

H

Hearse•11/23/23, 12:14 PM

huh thats weird.

anyone knows how to do full database backup with wrangler ?

Similar Threads

Similar Threads

Similar Threads