Our D1 database has been unavailable for the last 10mins withL: `D1_ERROR: Internal error while star

Our D1 database has been unavailable for the last 10mins withL:

D1_ERROR: Internal error while starting up D1 storage caused object to be reset.

D1_ERROR: Internal error while starting up D1 storage caused object to be reset.

D1_ERROR: Internal error while starting up D1 storage caused object to be reset.

D1_ERROR: Internal error while starting up D1 storage caused object to be reset.

rubenfOP•10/29/24, 8:42 PM

even the cloud console can't access it

rubenfOP•10/29/24, 8:42 PM

Are D1 databases beta/unfit for prod?

Rrubenf Our D1 database has been unavailable for the last 10mins withL: `D1_ERROR: Inter...

fry69•10/29/24, 8:45 PM

Which region/colo?

rubenfOP•10/29/24, 8:46 PM

@fry69 not sure actually, where do I find that info

rubenfOP•10/29/24, 8:46 PM

we have the database id

Rrubenf we have the database id

fry69•10/29/24, 8:56 PM

running_in_regionrunning_in_region via wrangler d1 info <dbname>wrangler d1 info <dbname>

fry69•10/29/24, 8:57 PM

rubenfOP•10/29/24, 8:58 PM

  Internal error: failed to get DB size and num_tables [code: 7500]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

  Internal error: failed to get DB size and num_tables [code: 7500]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

rubenfOP•10/29/24, 8:58 PM

the whole db is completely unavailable

1984 Ford Laser•10/29/24, 8:59 PM

syntax error?

Rrubenf ``` Internal error: failed to get DB size and num_tables [code: 7500] If ...

fry69•10/29/24, 8:59 PM

Worked for me in region EEUREEUR (creating a new D1 db and showing the info)

rubenfOP•10/29/24, 8:59 PM

that's using the CLI

rubenfOP•10/29/24, 8:59 PM

We have numerous D1 database, this one is failing, not the others

itsmatteomanf•10/29/24, 9:00 PM

Yeah, my best bet is that it's an issue of the specific D1 instance... can we ping somebody in the D1 team or escalate it, @Walshy?

1984 Ford Laser•10/29/24, 9:00 PM

https://github.com/cloudflare/workers-sdk/issues/4713
There's this that relates to comments in sql file, maybe relevant?

Rrubenf even the cloud console can't access it

Chaika•10/29/24, 9:00 PM

what's the id of the db and your account id?

Chaika•10/29/24, 9:00 PM

even if the dash doesn't load, should still let you see it

CChaika even if the dash doesn't load, should still let you see it

itsmatteomanf•10/29/24, 9:01 PM

a while back I had a db fail similarly, it wouldn't show it at all

rubenfOP•10/29/24, 9:01 PM

is it ok to share the id publicly ?

Chaika•10/29/24, 9:01 PM

yes

rubenfOP•10/29/24, 9:01 PM

cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanf•10/29/24, 9:01 PM

but yeah, db id and account id are fine to share

Iitsmatteomanf a while back I had a db fail similarly, it wouldn't show it at all

Chaika•10/29/24, 9:01 PM

must depend on type of failure, when my broke it (way back during the beta) just refused to load details

rubenfOP•10/29/24, 9:02 PM

We have moved everything from RDS to D1, and we're now a little bit concerned to say the least

CChaika must depend on type of failure, when my broke it (way back during the beta) just...

itsmatteomanf•10/29/24, 9:02 PM

mine had a storage issue on the node, fully collapsed and didn't show anywhere, it remained in the loading state

Rrubenf We have moved everything from RDS to D1, and we're now a little bit concerned to...

itsmatteomanf•10/29/24, 9:02 PM

let me escalate this for you

Rrubenf cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanf•10/29/24, 9:02 PM

Mind giving me the account id, too?

itsmatteomanf•10/29/24, 9:02 PM

That's the DB ID, right?

rubenfOP•10/29/24, 9:03 PM

correct, just gotta find out where to get our account id

Hugo•10/29/24, 9:03 PM

our account id is f3802b49118373bb0dd68e319c9be207

Iitsmatteomanf Yeah, my best bet is that it's an issue of the specific D1 instance... can we pi...

Walshy•10/29/24, 9:03 PM

dealing with a diff inc atm - p3 it pls

WWalshy dealing with a diff inc atm - p3 it pls

itsmatteomanf•10/29/24, 9:05 PM

Done!

HHugo our account id is f3802b49118373bb0dd68e319c9be207

itsmatteomanf•10/29/24, 9:05 PM

Thank you, escalated.

rubenfOP•10/29/24, 9:05 PM

itsmatteomanf•10/29/24, 9:07 PM

Might require D1 team manual intervention, but (won't guarantee it, of course, have no idea about internally and what the issue is) it's always been a full data recovery, if it needs to be rebuilt or something

rubenfOP•10/29/24, 9:08 PM

the other question that i have is regardless of issue, shouldn't time travel/recovery be possible so that we can do urgent operations if needed since when we read the blog posts there was redundancy

rubenfOP•10/29/24, 9:08 PM

we're fortunately not super super concerned with data loss as much as just the service being down and not possible to put back online by ourself

Rrubenf the other question that i have is regardless of issue, shouldn't time travel/rec...

itsmatteomanf•10/29/24, 9:11 PM

Time Travel requires the underlying instance to work, afaik... Time Travel is not a backup per se

itsmatteomanf•10/29/24, 9:11 PM

It's a point in time recovery

rubenfOP•10/29/24, 9:11 PM

My understanding is that it was diff of the WAL + snapshot from time to time no ?

itsmatteomanf•10/29/24, 9:11 PM

It's if you accidentally drop all tables :OMEGALUL:

Rrubenf My understanding is that it was diff of the WAL + snapshot from time to time no ...

itsmatteomanf•10/29/24, 9:12 PM

Yeah, but it's stored with the actual db, the backups are separate and are not accessible to the user directly

rubenfOP•10/29/24, 9:12 PM

ok so if we want to have production workload on this, we have to backup ourselves?

itsmatteomanf•10/29/24, 9:13 PM

afaik yes

itsmatteomanf•10/29/24, 9:13 PM

or have two synced D1s, they are always different objects, especially if they are different regions

Chaika•10/29/24, 9:13 PM

I mean no matter what platform you use, you should never trust their backups, do your own, different cloud

itsmatteomanf•10/29/24, 9:14 PM

And also, never trust any platform to be up 100% of the time

Chaika•10/29/24, 9:14 PM

well..

rubenfOP•10/29/24, 9:15 PM

Idk if I agree, the whole point of having multi AZ deployments is that you can trust services like aws to never have all regions down at the same time, and thus trust snapshot of dbs to be stored there

Rrubenf Idk if I agree, the whole point of having multi AZ deployments is that you can t...

Chaika•10/29/24, 9:15 PM

ok and someone compromises your account, then what?

Our D1 database has been unavailable for the last 10mins withL: `D1_ERROR: Internal error while star

Similar Threads

Similar Threads

Similar Threads