It started at 12PM UTC, I've been receiving these errors since then. D1 storage operation exceeded t

It started at 12PM UTC, I've been receiving these errors since then.
D1 storage operation exceeded timeout which caused object to be reset. / Network connection lost. / internal error

Ssz19 It started at 12PM UTC, I've been receiving these errors since then. D1 storage ...

TomK•10/29/24, 4:38 PM

I also encountered the same issue at the same time.

TTomK I also encountered the same issue at the same time.

sz19OP•10/29/24, 4:49 PM

is your d1 database located somewhere in asia?

Ssz19 is your d1 database located somewhere in asia?

TomK•10/29/24, 4:57 PM

yes

TomK•10/29/24, 4:59 PM

I'm not sure if this has anything to do with the maintenance program in the Singapore region?

TomK•10/29/24, 5:02 PM

TTomK Click to see attachment

fry69•10/29/24, 5:09 PM

That is still in the future if I read this correctly (UTC)

fry69•10/29/24, 5:10 PM

(should start in <1h)

fry69•10/29/24, 5:12 PM

The hiccups could of course be due to preparations for this maintenance or other things.

Ffry69 That is still in the future if I read this correctly (UTC)

TomK•10/29/24, 5:14 PM

Yes, it was future time, which is a very far-fetched reason indeed, but other than that I can't associate it with any other reason, since the code wasn't updated.:NotLikeThis:

sz19OP•10/29/24, 5:38 PM

Update: errors seems to be gone now on mine

sz19OP•10/29/24, 5:44 PM

There was about a solid 5 mins giving me this error:
Error: D1_ERROR: Internal error while starting up D1 storage caused object to be reset.
, then after that i no longer receive d1 errors. Hopefully it stays like this

rubenf•10/29/24, 8:41 PM

Our D1 database has been unavailable for the last 10mins withL: D1_ERROR: Internal error while starting up D1 storage caused object to be reset.D1_ERROR: Internal error while starting up D1 storage caused object to be reset.

rubenf•10/29/24, 8:42 PM

even the cloud console can't access it

rubenf•10/29/24, 8:42 PM

Are D1 databases beta/unfit for prod?

Rrubenf Our D1 database has been unavailable for the last 10mins withL: `D1_ERROR: Inter...

fry69•10/29/24, 8:45 PM

Which region/colo?

rubenf•10/29/24, 8:46 PM

@fry69 not sure actually, where do I find that info

rubenf•10/29/24, 8:46 PM

we have the database id

Rrubenf we have the database id

fry69•10/29/24, 8:56 PM

running_in_regionrunning_in_region via wrangler d1 info <dbname>wrangler d1 info <dbname>

fry69•10/29/24, 8:57 PM

rubenf•10/29/24, 8:58 PM

  Internal error: failed to get DB size and num_tables [code: 7500]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

  Internal error: failed to get DB size and num_tables [code: 7500]
  
  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

rubenf•10/29/24, 8:58 PM

the whole db is completely unavailable

1984 Ford Laser•10/29/24, 8:59 PM

syntax error?

Rrubenf ``` Internal error: failed to get DB size and num_tables [code: 7500] If ...

fry69•10/29/24, 8:59 PM

Worked for me in region EEUREEUR (creating a new D1 db and showing the info)

rubenf•10/29/24, 8:59 PM

that's using the CLI

rubenf•10/29/24, 8:59 PM

We have numerous D1 database, this one is failing, not the others

itsmatteomanf•10/29/24, 9:00 PM

Yeah, my best bet is that it's an issue of the specific D1 instance... can we ping somebody in the D1 team or escalate it, @Walshy?

1984 Ford Laser•10/29/24, 9:00 PM

https://github.com/cloudflare/workers-sdk/issues/4713
There's this that relates to comments in sql file, maybe relevant?

Rrubenf even the cloud console can't access it

Chaika•10/29/24, 9:00 PM

what's the id of the db and your account id?

Chaika•10/29/24, 9:00 PM

even if the dash doesn't load, should still let you see it

CChaika even if the dash doesn't load, should still let you see it

itsmatteomanf•10/29/24, 9:01 PM

a while back I had a db fail similarly, it wouldn't show it at all

rubenf•10/29/24, 9:01 PM

is it ok to share the id publicly ?

Chaika•10/29/24, 9:01 PM

yes

rubenf•10/29/24, 9:01 PM

cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanf•10/29/24, 9:01 PM

but yeah, db id and account id are fine to share

Iitsmatteomanf a while back I had a db fail similarly, it wouldn't show it at all

Chaika•10/29/24, 9:01 PM

must depend on type of failure, when my broke it (way back during the beta) just refused to load details

rubenf•10/29/24, 9:02 PM

We have moved everything from RDS to D1, and we're now a little bit concerned to say the least

CChaika must depend on type of failure, when my broke it (way back during the beta) just...

itsmatteomanf•10/29/24, 9:02 PM

mine had a storage issue on the node, fully collapsed and didn't show anywhere, it remained in the loading state

Rrubenf We have moved everything from RDS to D1, and we're now a little bit concerned to...

itsmatteomanf•10/29/24, 9:02 PM

let me escalate this for you

Rrubenf cbc9139f-7d2b-494d-b8ca-91875f273dda

itsmatteomanf•10/29/24, 9:02 PM

Mind giving me the account id, too?

itsmatteomanf•10/29/24, 9:02 PM

That's the DB ID, right?

rubenf•10/29/24, 9:03 PM

correct, just gotta find out where to get our account id

Hugo•10/29/24, 9:03 PM

our account id is f3802b49118373bb0dd68e319c9be207

Iitsmatteomanf Yeah, my best bet is that it's an issue of the specific D1 instance... can we pi...

Walshy•10/29/24, 9:03 PM

dealing with a diff inc atm - p3 it pls

WWalshy dealing with a diff inc atm - p3 it pls

itsmatteomanf•10/29/24, 9:05 PM

Done!

HHugo our account id is f3802b49118373bb0dd68e319c9be207

itsmatteomanf•10/29/24, 9:05 PM

Thank you, escalated.

rubenf•10/29/24, 9:05 PM

itsmatteomanf•10/29/24, 9:07 PM

Might require D1 team manual intervention, but (won't guarantee it, of course, have no idea about internally and what the issue is) it's always been a full data recovery, if it needs to be rebuilt or something

rubenf•10/29/24, 9:08 PM

the other question that i have is regardless of issue, shouldn't time travel/recovery be possible so that we can do urgent operations if needed since when we read the blog posts there was redundancy

rubenf•10/29/24, 9:08 PM

we're fortunately not super super concerned with data loss as much as just the service being down and not possible to put back online by ourself

Rrubenf the other question that i have is regardless of issue, shouldn't time travel/rec...

itsmatteomanf•10/29/24, 9:11 PM

Time Travel requires the underlying instance to work, afaik... Time Travel is not a backup per se

It started at 12PM UTC, I've been receiving these errors since then. D1 storage operation exceeded t

Similar Threads

Similar Threads

Similar Threads