EMERGENCY: Production database keeps going down
Our production database keeps going down with as an error message. This is a major issue.
The only resolution is to manually restart the compute endpoint each time.
Other context:
- us-east-2 endpoint
- Autoscaling is off
- Happened for the first time yesterday at 10:43 am PST
- Recently enabled logical replication
- Happened for the first time minutes after triggering an airbyte sync, but has happened 3+ times since then without us doing anything.
- Same day it started happening, neon had this status incident (https://neonstatus.com/aws-us-east-ohio/incidents/01HYDN1JG1MKS6HK9DS75HT358#01HYDN1JG16JHX8NHMNZ3D1FSC).
- Connection pooling is turned on and has been turned on for about a week. A google search of this error message brings up
We've tried contact neon support but have gotten no response even with priority support and marking the ticket serverity as "Critical: Production down"
The only resolution is to manually restart the compute endpoint each time.
Other context:
- us-east-2 endpoint
- Autoscaling is off
- Happened for the first time yesterday at 10:43 am PST
- Recently enabled logical replication
- Happened for the first time minutes after triggering an airbyte sync, but has happened 3+ times since then without us doing anything.
- Same day it started happening, neon had this status incident (https://neonstatus.com/aws-us-east-ohio/incidents/01HYDN1JG1MKS6HK9DS75HT358#01HYDN1JG16JHX8NHMNZ3D1FSC).
- Connection pooling is turned on and has been turned on for about a week. A google search of this error message brings up
pgBouncerWe've tried contact neon support but have gotten no response even with priority support and marking the ticket serverity as "Critical: Production down"
AWS - US East (Ohio) - us-east-2 Status
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.
