ap-southeast-1 Storage API very slow
Hey guys,
Is anyone experiencing performance issues in the ap-southeast-1 region with the Storage API, more specifically the /storage/v1/object/move endpoint? Requests are taking ~30s to complete, and the failure rate has increased as well. It's been like this for more than 24 hours now. There's nothing in the Supabase alerts page, so I wonder what might be going on.
16 Replies
The only mention here of ap-southeast-1 is https://discord.com/channels/839993398554656828/1419309397032701982/1419799213390172220 from 2 days ago with a user reporting slowness.
Nothing else by that name and no real reports of slow storage I can think of here or Github the past couple of days.
Are multiple users seeing this or just you?
If just you try a VPN or another device/network.
All of our customers are experiencing this issue. Does Supabase enforce any soft limits or throttling behind the scenes that could degrade performance? We're on the Pro plan and have exceeded the quota a long time ago, spend cap is turned off though
It would be best to ask support. I've not seen anything specific on storage throttling. If you have a lot of storage downloads I would certainly make sure you are taking advantage of caching and public buckets as much as possible.
Why are you doing lots of moves may I ask? Seems like a rare thing to need to do?
Let's just say I've inherited bad architectural decisions... customers can create documents based on templates, and when that happens the original template file is copied over to the document's bucket. There's a lot to revisit, but for now I need to get this working as is.
I've just noticed though that the UPDATE storage.objects statament Supabase does behind the scenes is taking more than 30s to complete. That seems to the obvious culprit now
@coloradointer Seems like RLS performance issue possibly.
What are the update and select policies? I might be able to offer you some quick ideas if you show them.
How many rows is storage.objects?
All of these apply to storage.objects and RLS except you can't control what indexes are there... https://supabase.com/docs/guides/database/postgres/row-level-security#rls-performance-recommendations
RLS is enabled but there are no policies. storage.objects has about 600k rows. What I really don't understand is that 2 days ago we had no such issues, and we haven't changed anything at all since then. We didn't have a massive influx of new files as well. I wonder if it's a new or updated trigger Supabase had added/changed, but then others would probably have the same issue too
There were some triggers added last week... So could have hit 2 days ago in production potentially.
https://github.com/supabase/storage/blob/master/migrations/tenant/0040-fix-prefix-race-conditions-optimized.sql
I have no idea if these would impact you or not.
Are you doing all uploads with service_role?
Otherwise you have to have RLS policies Look here to see when applied....
Otherwise you have to have RLS policies Look here to see when applied....
Mine says .... 9/22 for migration 40.
I would contact support and/or file an issue in supabase/storage.
That's exactly it! The fix-prefix-race-conditions-optimized migration ran 2 days ago and added the objects_update_cleanup trigger
As to why it is not impacting anyone else, is most don't have that many files and almost no one does .move. Or at least outside of some rare cases or a maintenance/change of their structure.
Thanks so much @garyaustin for pointing that out. I didn't know where else to look
I'll bring that up with Supabase
I would do both support AND file an issue in the repository.
That was my next question. Thanks again, really appreciated!
Just don't know which will move faster. Make sure on support to point out the migration. I had no idea there were new changes.
They are already working on a fix! https://github.com/supabase/storage/issues/769
GitHub
objects_update_cleanup trigger causes severe slowdowns for single...⚠️ Severe performance regression: single-row updates on storage.objects now take 30+ seconds due to the new trigger. Description Since the introduction of the objects_update_cleanup trigger on stor...