PostgreSQL won't start after cache was full. Error 403
Goodmorning, a few days ago I was downloading stuff on my unraid server and I left it for a few hours. When I came back, my cache was full and the Immich docker containers stopped. Now I cannot start again. Could you help me out please? Thanks in advance ❤️
14 Replies
:wave: Hey @WheelFire,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.I'm running Immich and PostgreSQL_Immich as docker containers so I don't have a docker-compose and .env files. The database however is just in my appdata share and Immich has access to it so I don't think that's the problem
I don't know where to find unraid docker logs (sorry I'm a noob) so I just copied the logs to a .txt file (I hope they are the same as the ones you need)
There's something wrong with the filesystem permissions on your postgres data folder
2025-02-17 11:22:53.707 CET [235] PANIC: could not write to log file 0000000100000004000000BF at offset 4898816, length 8192: No space left on deviceAnd the disk is still full
I used the mover in Unraid to clean my cache disk so there should be enough space left again (around 140 gb). Also I used the 'new permissions' tool on the appdata share where my database is located. Those still didn't fix it unfortunately
Is there maybe another tool I could use? Also because the container won't start again, those logs are from 2 days ago when it crashed (they are also the latest available logs)
You're saying that the container doesn't output any logs at all when you try to start it now? That seems very odd
yes, I get an "execution error, error 403" and then it stays on 'stopped'
I imagine something got messed up in unraid when your array ran out of space
Reboot the machine and see if it still persists after that
I rebooted and it worked again! I'm sorry for not trying that before but I always had issues when rebooting unraid (i think i fixed it now)
I still saw this in the logs: [29] LOG: invalid record length at 4/BF4ACEC8: expected at least 24, got 0
however, after that it said checkpoint complete and that the database system is ready to accept connections so I think it's all right
Thanks for your time ❤️
Do you still get that error every time you restart?
the invalid record length?
Yes
no not anymore now
Postgres likely correct it, sounds like you're all good 🙂
Great! Thanks again