Production Database in Crash Loop - Cannot Self-Recover
🔢DB Connections🤷other🟢SQL
What happened:
A team member accidentally inserted millions of records into a new table, which overwhelmed the database. It crashed and is now stuck in an infinite crash loop.
Current state:
Database crashes every ~20 seconds during recovery
Logs show: "database system was interrupted while in recovery... This probably means that some data is corrupted"
Error: FATAL: 57P03 - the database system is not accepting connections
Cannot access PITR in dashboard - won't load
Cannot connect at all
Production app completely down for hours
Logs pattern (repeating every 20s):
LOG: database system was not properly shut down; automatic recovery in progress
LOG: database system was interrupted while in recovery... data is corrupted
(crash, restart, repeat)
LOG: database system was not properly shut down; automatic recovery in progressLOG: database system was interrupted while in recovery... data is corrupted(crash, restart, repeat)
What I need:
Immediate escalation to infrastructure team
Point-in-time restore to before the insert, OR
Manual intervention to skip corrupted WAL segments
Support ticket says 2-day response time but this is a production outage affecting our business. Any help escalating would be hugely appreciated.
A team member accidentally inserted millions of records into a new table, which overwhelmed the database. It crashed and is now stuck in an infinite crash loop.
Current state:
Database crashes every ~20 seconds during recovery
Logs show: "database system was interrupted while in recovery... This probably means that some data is corrupted"
Error: FATAL: 57P03 - the database system is not accepting connections
Cannot access PITR in dashboard - won't load
Cannot connect at all
Production app completely down for hours
Logs pattern (repeating every 20s):
LOG: database system was not properly shut down; automatic recovery in progress
LOG: database system was interrupted while in recovery... data is corrupted
(crash, restart, repeat)
LOG: database system was not properly shut down; automatic recovery in progressLOG: database system was interrupted while in recovery... data is corrupted(crash, restart, repeat)
What I need:
Immediate escalation to infrastructure team
Point-in-time restore to before the insert, OR
Manual intervention to skip corrupted WAL segments
Support ticket says 2-day response time but this is a production outage affecting our business. Any help escalating would be hugely appreciated.
