I ran a recursive query and now there is no more space on device

I am getting this error: 2024-03-29 22:10:54.413 UTC [8] FATAL: could not write lock file "postmaster.pid": No space left on device but i can't access the database with psql or anything to try and remove the uneccessary data. This is resulting in downtime for us.
Solution:
Should be up to 100 GB
Jump to solution
15 Replies
Percy
Percy3mo ago
Project ID: N/A
Brody
Brody3mo ago
hey @themkrage what is the current size of your volume?
themkrage
themkrage3mo ago
50GB It looks like the recusrive pg query i ran filled it. It's not like the database has 50GB's of content, just that one query appears to have filled the container
No description
themkrage
themkrage3mo ago
i don't see an option to increase the cap, and im not sure how to clear the "in-progress" query and data i can't connect to the database either, so i can't run VACUUM, which i think would fix this
Brody
Brody3mo ago
Okay ill do my best to get a team member here asap, in the mean time please provide your project id
themkrage
themkrage3mo ago
/98f66160-ce86-4b15-a73b-cab912256208
themkrage
themkrage3mo ago
Railway
404 - Page not found
Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.
angelo
angelo3mo ago
Hello, working on expanding your cap now One sec
themkrage
themkrage3mo ago
thank you!
angelo
angelo3mo ago
Okay, try now
Solution
angelo
angelo3mo ago
Should be up to 100 GB
themkrage
themkrage3mo ago
thank you! able to connect. now running VACUUM thank you. Running vacuum brought our disk usage back to normal expected level For future reference, is there no way to access a database that has hit the disk usage cap?
Brody
Brody3mo ago
correct postgres more or less locks itself up when it runs out of disk
themkrage
themkrage3mo ago
is there some internal safe guard we could add to prevent this?
Brody
Brody3mo ago
there is nothing that i am personally aware of