Cloudflare Developers•3y ago

id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e181305f10

id : df608b9e06cca37a7fc399c8ca86eab4
database : 0f3e37af-ecd2-4e73-9766-72e181305f10

YYokoWasis id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e1813...

Cyb3r-Jak3•11/22/23, 12:59 AM

CC @Matt

YokoWasisOP•11/22/23, 1:03 AM

anyone experiencing the same, or is it just me ?

Cyb3r-Jak3•11/22/23, 1:09 AM

There were some reports earlier (d1-database) but they rolled back

Unsmart•11/22/23, 1:17 AM

My table that was broken seems like it still is so

maybe its not fully rolled back yet or they redid it not sure. Not too worried as I wouldnt use anything production with a beta product and it happened on a table I dont even use and I dont know why I made KEKW

YokoWasisOP•11/22/23, 1:23 AM

anyone can give an eta about this ?

Unsmart•11/22/23, 1:24 AM

It's a beta product so probably whenever they are online again tomorrow it will be looked at

YokoWasisOP•11/22/23, 1:31 AM

YYokoWasis id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e1813...

Matt Silverlock•11/22/23, 1:46 AM

We’re investigating but can’t promise this will be resolved immediately

UUnsmart My table that was broken seems like it still is so 🤷 maybe its not fully rolled...

Matt Silverlock•11/22/23, 1:46 AM

Fully rolled back. A single table being broken is something else. The issue prevented the DB from loading. There is no such thing as a “partial” load - we don’t load incremental pages. I would not associate with today’s issue.

Unsmart•11/22/23, 1:48 AM

I meant database not table in that message... I dont think it even has any tables but not really sure since I dont use that database at all and dont remember making it

Unsmart•11/22/23, 1:51 AM

as in I still get this and it hasnt changed since the rollback (specifically the last database):

YYokoWasis id : df608b9e06cca37a7fc399c8ca86eab4 database : 0f3e37af-ecd2-4e73-9766-72e1813...

Matt Silverlock•11/22/23, 2:17 AM

We see this database working now.

MMatt Silverlock We see this database working now.

YokoWasisOP•11/22/23, 2:35 AM

thanks, it's working now

YokoWasisOP•11/22/23, 2:37 AM

is this happened for all of d1, or specific few databases ? should I create a database as a mirror in case this happened again ?

YYokoWasis is this happened for all of d1, or specific few databases ? should I create a da...

Matt Silverlock•11/22/23, 2:40 AM

We saw < 10 databases impacted here.

James•11/22/23, 2:54 AM

It's not the first time folks have hit this same error message before. I'm not sure if the root cause is the same or not, but especially for issues like this that externally seem to be recurring ones, it would be really appreciated if a status page could be put up in future. It saves a lot of the reactive back and forth that happens here, and only improves the transparency we all know and love Cloudflare for MeowHeartCloudflare

MMatt Silverlock We saw < 10 databases impacted here.

YokoWasisOP•11/22/23, 2:59 AM

thanks, if this is the case I will create a mirror for my database as a backup.

YokoWasisOP•11/22/23, 3:02 AM

how to download full backup of database ?

James•11/22/23, 3:53 AM

Retro status page went up: https://www.cloudflarestatus.com/incidents/2j7wtjt39ypl

Small number of D1 databases failed to execute queries

James•11/22/23, 3:53 AM

Thanks all MeowHeartCloudflare

MMatt Silverlock We see this database working now.

YokoWasisOP•11/22/23, 4:22 AM

it's down again

YokoWasisOP•11/22/23, 5:10 AM

anyone knows how to do full database backup with wrangler ?

YYokoWasis anyone knows how to do full database backup with wrangler ?

James•11/22/23, 5:12 AM

You can’t yet directly with wrangler, but you can use this great community solution: https://github.com/nora-soderlund/cloudflare-d1-backups

GitHub

GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a ...

A package to dump a copy of your experimental D1 database through a worker. - GitHub - nora-soderlund/cloudflare-d1-backups: A package to dump a copy of your experimental D1 database through a worker.

federo•11/22/23, 11:39 AM

can anyone help me understand how is this viable if a database can only hold 2gb of data?

zegevlier•11/22/23, 11:42 AM

(not a team member, just from my knowledge as an outsider)
First: The goal is still to increase this during the beta. Second: With D1, you can have a lot of seperate databases, all with a limit of 2gb. If your application can support sharding the DB like this, there isn't really a limit to the amount of data you can store. To me it seems perfect for a lot of SaaS applications, where each customer, or even each project, would have their own database.

federo•11/22/23, 11:44 AM

so if a client would then need to make a query, the query would have to first look at which database to look into and perhaps query the data across databases?

Ffedero so if a client would then need to make a query, the query would have to first lo...

Max (@rozenmd)•11/22/23, 11:46 AM

a certain issue manager that many folks dislike handles multi-tenancy that way, yeah - one db per customer.

you keep a highly available store mapping something (customerId, a subdomain, etc) to the database ID, then pass the databaseID to your client (we're working on making the d1-api support this nicely)

Max (@rozenmd)•11/22/23, 11:48 AM

migrations are also a pain atm with this model: https://github.com/cloudflare/workers-sdk/issues/4471

federo•11/22/23, 11:48 AM

but if a customers needs way more than 2gb of data, then do you see this viable?
I am trying to setup a flow where a js script on websites sends events data to a cloudflare worker, from which I am trying to understand how to best forward it to a database

Max (@rozenmd)•11/22/23, 11:49 AM

for events data d1 is going to have a bad time, you might want to look into https://developers.cloudflare.com/analytics/analytics-engine/

Ffedero but if a customers needs way more than 2gb of data, then do you see this viable?...

zegevlier•11/22/23, 11:49 AM

also see: d1-database

federo•11/22/23, 11:52 AM

analytics engine is not feasible either with a retention period of only 3 months

Zzegevlier also see: https://discord.com/channels/595317990191398933/992060581832032316/117...

federo•11/22/23, 12:04 PM

it seems like Hyperdrive is just for reads not writes

GGustavo One idea would be to aggregate and move to somewhere else

federo•11/22/23, 12:05 PM

how do you move it to somewhere else? I am struggling to understand how to take payloads coming into workers and load them into a solid database

Ffedero it seems like Hyperdrive is just for reads not writes

kian•11/22/23, 12:06 PM

Writes will just passthrough the caching and still take advantage of the Cloudflare backbone.

GGustavo I mean write to Analytics Engine, then have a cron job run each month to aggrega...

Rob M.•11/22/23, 12:29 PM

But what about WAE sampling? Looks like we can lose data in some cases.

RRob M.But what about WAE sampling? Looks like we can lose data in some cases.

Max (@rozenmd)•11/22/23, 12:46 PM

if you need every single row, i would just use postgres

Ffedero it seems like Hyperdrive is just for reads not writes

Matt Silverlock•11/22/23, 1:28 PM

Hyperdrive doesn’t restrict you. Curiously: why do you think it’s just reads?

Nnick_jordan When running a query from the UI Console I'm getting the following error... ```...

Max (@rozenmd)•11/22/23, 3:07 PM

That looks like an alpha database - the current default backend for D1 performs around 10-11x faster for writes, 20x faster for reads: [https://blog.cloudflare.com/d1-turning-it-up-to-11/]

ajgeiss0702•11/22/23, 5:25 PM

in the dashboard, is there a plan to add rows read/written so I can have an idea on what my bill would be? I know I can see it for the past week by going to the per-db metrics, but I would like to be able to see the total for the month since that what i'll be billed for once billing gets enabled