Cloudflare Developers•3y ago

We see this database working now.

MMatt Silverlock We see this database working now.

YokoWasis•11/22/23, 2:35 AM

thanks, it's working now

YokoWasis•11/22/23, 2:37 AM

is this happened for all of d1, or specific few databases ? should I create a database as a mirror in case this happened again ?

YYokoWasis is this happened for all of d1, or specific few databases ? should I create a da...

Matt SilverlockOP•11/22/23, 2:40 AM

We saw < 10 databases impacted here.

James•11/22/23, 2:54 AM

It's not the first time folks have hit this same error message before. I'm not sure if the root cause is the same or not, but especially for issues like this that externally seem to be recurring ones, it would be really appreciated if a status page could be put up in future. It saves a lot of the reactive back and forth that happens here, and only improves the transparency we all know and love Cloudflare for MeowHeartCloudflare

MMatt Silverlock We saw < 10 databases impacted here.

YokoWasis•11/22/23, 2:59 AM

thanks, if this is the case I will create a mirror for my database as a backup.

YokoWasis•11/22/23, 3:02 AM

how to download full backup of database ?

James•11/22/23, 3:53 AM

Retro status page went up: https://www.cloudflarestatus.com/incidents/2j7wtjt39ypl

Small number of D1 databases failed to execute queries

James•11/22/23, 3:53 AM

Thanks all MeowHeartCloudflare

MMatt Silverlock We see this database working now.

YokoWasis•11/22/23, 4:22 AM

it's down again

YokoWasis•11/22/23, 5:10 AM

anyone knows how to do full database backup with wrangler ?

YYokoWasis anyone knows how to do full database backup with wrangler ?

James•11/22/23, 5:12 AM

You can’t yet directly with wrangler, but you can use this great community solution: https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

federo•11/22/23, 11:39 AM

can anyone help me understand how is this viable if a database can only hold 2gb of data?

zegevlier•11/22/23, 11:42 AM

(not a team member, just from my knowledge as an outsider)
First: The goal is still to increase this during the beta. Second: With D1, you can have a lot of seperate databases, all with a limit of 2gb. If your application can support sharding the DB like this, there isn't really a limit to the amount of data you can store. To me it seems perfect for a lot of SaaS applications, where each customer, or even each project, would have their own database.

federo•11/22/23, 11:44 AM

so if a client would then need to make a query, the query would have to first look at which database to look into and perhaps query the data across databases?

Ffedero so if a client would then need to make a query, the query would have to first lo...

Max (@rozenmd)•11/22/23, 11:46 AM

a certain issue manager that many folks dislike handles multi-tenancy that way, yeah - one db per customer.

you keep a highly available store mapping something (customerId, a subdomain, etc) to the database ID, then pass the databaseID to your client (we're working on making the d1-api support this nicely)

Max (@rozenmd)•11/22/23, 11:48 AM

migrations are also a pain atm with this model: https://github.com/cloudflare/workers-sdk/issues/4471

federo•11/22/23, 11:48 AM

but if a customers needs way more than 2gb of data, then do you see this viable?
I am trying to setup a flow where a js script on websites sends events data to a cloudflare worker, from which I am trying to understand how to best forward it to a database

Max (@rozenmd)•11/22/23, 11:49 AM

for events data d1 is going to have a bad time, you might want to look into https://developers.cloudflare.com/analytics/analytics-engine/

Ffedero but if a customers needs way more than 2gb of data, then do you see this viable?...

zegevlier•11/22/23, 11:49 AM

also see: d1-database

federo•11/22/23, 11:52 AM

analytics engine is not feasible either with a retention period of only 3 months

Zzegevlier also see: https://discord.com/channels/595317990191398933/992060581832032316/117...

federo•11/22/23, 12:04 PM

it seems like Hyperdrive is just for reads not writes

GGustavo One idea would be to aggregate and move to somewhere else

federo•11/22/23, 12:05 PM

how do you move it to somewhere else? I am struggling to understand how to take payloads coming into workers and load them into a solid database

Ffedero it seems like Hyperdrive is just for reads not writes

kian•11/22/23, 12:06 PM

Writes will just passthrough the caching and still take advantage of the Cloudflare backbone.

GGustavo I mean write to Analytics Engine, then have a cron job run each month to aggrega...

Rob M.•11/22/23, 12:29 PM

But what about WAE sampling? Looks like we can lose data in some cases.

RRob M.But what about WAE sampling? Looks like we can lose data in some cases.

Max (@rozenmd)•11/22/23, 12:46 PM

if you need every single row, i would just use postgres

Ffedero it seems like Hyperdrive is just for reads not writes

Matt SilverlockOP•11/22/23, 1:28 PM

Hyperdrive doesn’t restrict you. Curiously: why do you think it’s just reads?

Nnick_jordan When running a query from the UI Console I'm getting the following error... ```...

Max (@rozenmd)•11/22/23, 3:07 PM

That looks like an alpha database - the current default backend for D1 performs around 10-11x faster for writes, 20x faster for reads: [https://blog.cloudflare.com/d1-turning-it-up-to-11/]

ajgeiss0702•11/22/23, 5:25 PM

in the dashboard, is there a plan to add rows read/written so I can have an idea on what my bill would be? I know I can see it for the past week by going to the per-db metrics, but I would like to be able to see the total for the month since that what i'll be billed for once billing gets enabled

Aajgeiss0702 in the dashboard, is there a plan to add rows read/written so I can have an idea...

Max (@rozenmd)•11/22/23, 5:31 PM

That's a good point - I'll add it to our backlog

MMax (@rozenmd)That's a good point - I'll add it to our backlog

ajgeiss0702•11/22/23, 5:38 PM

ty!

YokoWasis•11/23/23, 12:05 AM

Let's hope today D1 won't be down like yesterday. I made 4 replica of the database. It would be absurd of all of them going down.

YokoWasis•11/23/23, 12:52 AM

My D1 database said too many queue operation. Any recommendation ? is D1 is not feasible for many users ?

Tesbarmix•11/23/23, 12:53 AM

Hello

YokoWasis•11/23/23, 12:59 AM

What about kv ? does kv can handle R/W requests better than d1 ?

YYokoWasis My D1 database said too many queue operation. Any recommendation ? is D1 is not ...

Max (@rozenmd)•11/23/23, 8:54 AM

what's your use case?

MMax (@rozenmd)what's your use case?

YokoWasis•11/23/23, 9:17 AM

student's national exam.

YYokoWasis student's national exam.

Max (@rozenmd)•11/23/23, 9:20 AM

as in, are you using D1 as a single database for an API that your frontend talks to? how many requests per second?

MMax (@rozenmd)as in, are you using D1 as a single database for an API that your frontend talks...

YokoWasis•11/23/23, 9:54 AM

Yes, worker as a backend api. About 1k req per second.

Hearse•11/23/23, 11:17 AM

wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Could not read file: ./drizzle/0000_short_lockheed.sql

  ENOENT: no such file or directory, open
  '/Users/administrator/backend/project/drizzle/0000_short_lockheed.sql'
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

wrangler d1 execute 'cloudflare-db' --local --file='./drizzle/0000_short_lockheed.sql'

✘ [ERROR] Could not read file: ./drizzle/0000_short_lockheed.sql

  ENOENT: no such file or directory, open
  '/Users/administrator/backend/project/drizzle/0000_short_lockheed.sql'
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

Hearse•11/23/23, 11:18 AM

Why is it doing this when following https://github.com/drizzle-team/drizzle-orm/blob/main/examples/cloudflare-d1/README.md

GitHub

drizzle-orm/examples/cloudflare-d1/README.md at main · drizzle-team...

TypeScript ORM that feels like writing SQL. Contribute to drizzle-team/drizzle-orm development by creating an account on GitHub.