Complete DB Outage 2 days in a row

Hey,

I've had a complete DB outage in our project 2 days in a row. I've opened a ticket, but just curious if others have seen this.

I've had to restart everything in the supabase console to get it back up and running. We go from 1% of our IOPs limit to suddenly 90% and the app stops working. i cant find any metrics that correlate to this.

garyaustin•11/29/25, 9:32 PM

There is nothing in the logs (API Gateway) common at point of increase?
Any very slow queries in the query report?

garyaustin•11/29/25, 9:32 PM

Did they occur at same time?

ArotonOP•11/30/25, 8:33 PM

The thing I think it might have been is we were doing getUser requests in an api instead of jwt user validation so our auth requests were VERY high.

The weird thing to me is there are no metrics I can find that indicate anything was elevated.

For example all the DB metrics, the IOPs was like 2-3 and then suddenly the overall disk IO usage was "maxed out"

I think this is a supabase metrics issue tbh. as far as I can see the only way to see the disk IO usage, which is what you get throttled on, is to updgrade to the $600/month plan? Which seems wild to me.

IOPS is always extremely low in our app but apparently IO usage is high.

ArotonOP•11/30/25, 8:34 PM

Ill wait on my ticket with them, because at this point their metrics are next to worthless for actually root causing things.

ArotonOP•11/30/25, 8:35 PM

All DB metrics were very low usage. ram, cpu, iops, but somehow our disk usage jumped from 1% to 90% in a single day

garyaustin•11/30/25, 8:35 PM

How many users do you have? What size plan are you currently?

ArotonOP•11/30/25, 8:36 PM

100 daily active and on the small compute.

DB is only 1gb

garyaustin•11/30/25, 8:37 PM

Any chance the getUser calls were being falsely generated by a code bug?

garyaustin•11/30/25, 8:38 PM

100 users is not all that match. And I assume you meant $60 not $600 which is why I asked about size of your current plan.

ArotonOP•11/30/25, 8:38 PM

Nah, it was in middleware in my vercel API I was calling the Auth api instead of just doing it inside the API.

I migrated that to just use the app JWT key to decrypt the JWT.

Which lowered Auth calls dramatically. My issue is there really wasn't any indicator

ArotonOP•11/30/25, 8:38 PM

ArotonOP•11/30/25, 8:38 PM

I'm talking about this graph, which seems to be the metric you get throttled off of

ArotonOP•11/30/25, 8:38 PM

requires the $600 plan

ArotonOP•11/30/25, 8:38 PM

Its also possible the IOPs graph is just wrong too

garyaustin•11/30/25, 8:39 PM

I have no comment on the metrics charts. Not really seen any indication they are flawed from users, but certainly not always helpful debugging.

garyaustin•11/30/25, 8:39 PM

What feature requires a $600 plan?

ArotonOP•11/30/25, 8:39 PM

The disk IO usage graph

ArotonOP•11/30/25, 8:40 PM

The detailed one. You can see this one in the infrastructure tab (which is very buggy)

ArotonOP•11/30/25, 8:41 PM

No traffic change occured on those 2 days. Just randomly super high usage which is why I was wondering if anyone else saw it

ArotonOP•11/30/25, 8:41 PM

Anyways I have a ticket open... nothing was "out of the normal" so I don't know what happened

ArotonOP•11/30/25, 8:41 PM

ONly thing I can assume is the graphs are buggy or don't track the auth schema or something

ArotonOP•11/30/25, 8:42 PM

I did do a bunch of optimizations ... but my real concern is I have no idea when I actually need to upgrade bc I'm apparently being tracked on metrics invisible to me

ArotonOP•11/30/25, 8:42 PM

Bc from all the graphs I'm using < 1% of my resources, until suddenly my app is shut off

garyaustin•11/30/25, 8:43 PM

Hopefully they can help you.
Please report back if you find anything useful so it may help others.

garyaustin•11/30/25, 8:47 PM

There is also this for gathering more detail...
https://supabase.com/docs/guides/telemetry/metrics

ArotonOP•11/30/25, 9:18 PM

Ya, I was going to check out hosting costs for some of these platforms. I need to set up alarms anyways. Hopefully not too bad.

At least a supabase restart fixes the issue we were having

ArotonOP•12/2/25, 6:13 PM

So, support was not helpful at all.

Just said it was the io graph that is clearly broken.

I went ahead and set up grafana cloud metrics. What i've found is the supabase metrics in observability are just flat out wrong.

My IOPS is much higher than what it says.

And after the outages I did a TON of optimizations and it still 10x higher than what supabase is reporting. So it could have easily been one of these things pre optimization.

Guess I just assumed the metrics weren't entirely wrong.

So moral of the story is don't rely on supabase metrics

ArotonOP•12/2/25, 7:36 PM

Another note. Now that I have my metrics loaded and can see detailed usage. It is more likely that realtime connections are causing memory thrash.

Worth noting you need a LOT of memory if you are going to run these. I'm swapping over usage of them to a tanstack invalidation/polling solution instead

ArotonOP•12/2/25, 7:46 PM

I've just pushed the change out.

This all makes more sense in theory anyways. 100% swap memory in use -> memory thrash -> unresponsive.

Unfortunately the only way to see this is an external data provider. The memory usage graph inside supabase indicates everything is ok and there are plenty of resources.

garyaustin•12/2/25, 7:49 PM

Were you using postgres_changes or broadcast from DB with private channels?

Complete DB Outage 2 days in a row

Similar Threads

Similar Threads

Similar Threads