Postgres not starting anymore on Unraid
Upgrade to 1.133 went fine (with changed repository for postgres). Update to 1.133.1 also went fine, as did the postgres update yesterday. I confirmed, everything was OK after each update. This morning Immich would not start and I found Postgres to be in a condition, where it does not start. Upon checking the logs, I found that last night the server shut down. Find attached the errors in the log. Any help greatly appreciated, since databases are not my forte.
78 Replies
:wave: Hey @Mr.Penguin | DXP4800+ | Unraid,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Most likely unraid has moved the files off the cache drive / damaged them in some way
If you have any movers etc you will need to explore as to what that may have caused
Thanks, will look into that.
There is a mover scheduled daily for 03:40, unfortunately the logging for the Mover was disabled. But the incident (shutdown of postgres) happened around 20:00 last night. So I don't see a connection there.
Is there a way I can fix this or is it time for back up?
I am also using unraid and did what you did. Did you notice that the mobile app lost its automatic backup state (zero objects everywhere, only manual upload works). Looks like it is a symptom that I am about to get into the same trouble as you
Yep, everything is on zero. But the pictures are all there. They must be cached somehow.
looks like your situation is worse than mine. see the thread above
Yeah, just read it. Good, you got it sdolved. My problem is, that postgres is missing some files. Need to look into that later tonight.
But are you certain that your mover works for postgreqsql data? Because usually the postgres data goes straight to cache, not to a cache->array array. Unless you did that manually. And if so - why?
No, I am not sure, just looked into it briefly.
I am not good with database stuff and worse with postgres.
are you using the docker compose extension way from official docs or SpaceInvaderOne's way with two separate images ?
I am using the Docker, that unraid provides for both immich and postgres.
unraid provides none, its people who create different templates. could you link or screenshot them?
I use the immich container by imagegenius and the container for postgresql15 by postgres. The immich-postgresql container did not exist, when I installed immich.
so what are you referring to as postgres update in your initial message?
The container postgresql15 had an update available yesterday. It showed up some time after the one for immich 1.133.1, which I ran first
I do not think immich supports postgres15
Did you manually install a vecror extension there?
https://immich.app/docs/administration/postgres-standalone
It worked fine until yesterday. I changed the repository for postgres15 to vectorchord, which worked just fine.
did you manually install a vector extension to your previous PG instance?
The postgres update yesterday succeeded as well. Just this morning I saw thze problem.
what is postgres major version of the new image that you put there? 14 or 15 or 16?
I don't remember, if I installed it manually, but postgres15 worked with pgvecto.rs for more than a year.
postgres is still on 15
The postgres update yesterday was not a version upgrade.
what do you mean by postgres15 worked with pgvecto.rs for more than a year? where did you get pgvecto.rs extension from before if you used the official postgres 15 image?
As I said, I can't t remember. I think, there was a disruptive update to Immich, where I installed it. But this was ~ a year ago.
So, you used a manually set up postgres instance with pgvector.rs, but on the update you used a pre-built image. The migration might me incompatible that way. What I recommend:
try running your previous postgres instance up (if the regular postgres updates did not break immich, the extension might be still "alive" in a volumne)
shut off the immich
if the database starts, launch the immich on a pre-33 version
see if it works
If it does, try first migrating to prebuild vecto.rs image, and THEN to the vectorchord one
It might have been v1.95.0. This update required users to upgrade the pgvecto.rs extension used by the PostgreSQL database from version 0.1.11 to 0.2.0.
OK, I can try that, but not right now, I am at work. Thanks for all your input for now.
My approach would have been to go hunting for the missing files and put them back where they belong.
the main goal now is to try to get up your previous immich state. I am always certain no files are missing, its just incompitablility on some level
Because if so, every update of your postgres15 would break the database (if some data was not in a volumne)
from there it would be easier to carefully migrate the database to a proper instance
But I am not sure, we are on the same page. I am still using the same postgres container that I originally installed when I first installed Immich.
Yeah, I re-read this. Look like the migration was succesful indeed
this makes it less obvious
you say you are still using the same postgres container. What did you mean by "the same"? what is the image there now, a vectorchord one or a default one?
for the update ti Immich 1.133 I changed the repository from the one with pgvecto.rs to Vectorchord: ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0
I used Immich for 2 days after that, all was nice and dandy
was the one with pgvecto.rs version 15 or 14? You said your initial major version was 15
Did I understand it correctly that you had:
1. An official postgres15 image
2. At some point you changed that image to postgresXX-pgvecto.rs prebuilt image?
3. to update to 1.133 you have changed the image to ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0
I had and have 15 since first install
this one is 14 ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0
1.) yes
2.) I never changed an image of postgres afaik.
3. ghcr.io/immich-app/postgres:15-vectorchord0.3.0-pgvectors0.3.0
did you manually check for global/ files via unraid interface like this?
and here is the reference of how my file structure looks
you can find the location in docker menu
Because I googled that postgres error and sometimes it is a permissions issue they say
Let me do that real quick
What am I looking for in /global?
The permissons in /global are all -rw-----------
and my file structure is identical with yours
The difference is, that 3 dirs go from cache→disk, the rest goes straight to disk. The 3 are: base, global and pg_wal
I will be in a meeting for the next 2-3 hours. Will check in after that.
see logs in your first message
it is strange that it is on disk while you have a cache. Did you mount a share to postgres data instead of /user/appdata?
Also, if using HDD you had to set up another environment variable (see the changelog in github). In your case. cache and disk are messed up. I recommend just mounting the postgres in your appdata, not on a share and definetly not on an array share.
You can copy your mounted folder to /user/appdata via unraid interface and try mounting from there
pg_filenode_map exists with -rw---------
good news! some permissions issue. But still kinda messed up. I would first move that folder from a share to appdata (the whole folder that goes mounted to your postgres instance)
can't find postmaster.pid though
I would first move that folder ← can you be more specific please. Are you talking about /global?
ah, postmaster.pid is on ../appdata/postgresql15, but pg_filenode_map is in /global
-rw------- 1 999 users 512 Dec 13 2023 pg_filenode.map
-rw------- 1 999 999 94 May 26 00:03 postmaster.pid
do these look right to you?
no, I am talking about the folder that you see as mounted into the postgres container in your docker menu
I'm afraid, I can't follow
there is: /var/lib/postgresql/data → /mnt/cache/appdata/postgresql15
as volume mapping
I'm afraid, my install is kind of messed up
open console
cp -R /mnt/cache/appdata/postgresql15 /mnt/user/appdata/
then tap edit on the postgres container, and change the path to /mnt/user/appdata/postgresql15/
this is probably not gonna fix your issue as the mover might have broke the instance but at least you are going to have a proper destination for your postgres volume. But we also have hope for fixing the permisiions
as it should be, you can see in the logs that the postmaster is above
postgresql started up again 🙂
but immich does not come up
after what?
oh, postgres stopped again
after the cp -R and editing the path
can you share the logs
then, share the ls -la of new destionation to see the owner. Because the data is rw in my case
oot@Tower:/mnt/user/appdata# ls -al postgresql15
total 72
drwx------ 1 999 users 32 May 26 20:22 ./
drwxrwx--- 1 nobody users 534 May 26 05:46 ../
-rw------- 1 999 users 3 Dec 13 2023 PG_VERSION
drwx------ 1 999 users 10 Jun 7 2024 base/
drwx------ 1 999 users 16 May 26 00:04 global/
drwx------ 1 999 999 8192 May 23 09:00 log/
drwx------ 1 999 users 10 Dec 13 2023 pg_commit_ts/
drwx------ 1 999 users 10 Dec 13 2023 pg_dynshmem/
-rw------- 1 999 users 4821 Dec 13 2023 pg_hba.conf
-rw------- 1 999 users 1636 Dec 13 2023 pg_ident.conf
drwx------ 1 999 users 84 May 26 05:08 pg_logical/
drwx------ 1 999 users 48 Dec 13 2023 pg_multixact/
drwx------ 1 999 users 10 Jul 15 2024 pg_notify/
drwx------ 1 999 users 10 Dec 13 2023 pg_replslot/
drwx------ 1 999 users 10 Dec 13 2023 pg_serial/
drwx------ 1 999 users 10 Dec 13 2023 pg_snapshots/
drwx------ 1 999 users 10 May 26 00:03 pg_stat/
drwx------ 1 999 users 10 Dec 13 2023 pg_stat_tmp/
drwx------ 1 999 users 26 Feb 4 09:05 pg_subtrans/
drwx------ 1 999 users 10 Dec 13 2023 pg_tblspc/
drwx------ 1 999 users 10 Dec 13 2023 pg_twophase/
drwx------ 1 999 999 67 May 25 08:35 pg_vectors/
drwx------ 1 999 users 48 May 23 12:28 pg_wal/
drwx------ 1 999 users 138 Feb 4 09:05 pg_xact/
-rw------- 1 999 users 88 Dec 13 2023 postgresql.auto.conf
-rw------- 1 999 users 29549 Dec 13 2023 postgresql.conf
-rw------- 1 999 users 87 May 26 20:22 postmaster.opts
from my perspective this looks like bad news: a corruption. Maybe caused by shutdown. Try the integrity thing in unraid, I guess that is what is was made for:)
I mean parity
so run a parity check?
yeah and according to chatgpt if some files are corrupted you might get an option to restore it
I also think immich had automatic database backups, but still it will require a fresh reinstall of immich. Let me check for a second
for now, lets start with parity check
I am thinking of starting over clean, as I know a little biit more about Unraid as I did, when I first installed immich 15 months ago. Would I be able to use the backup of my db in a fresh setup?
but before that. shutdown both containers
you will and this is definetly a nice option
if you still have planned to do so, I guess the best option is to restore the backup
I am glad to help you with a clean unraid install
OK, shutdown both containers, then run parity check. That will take until the morning. If that does not work, I will start over and use immich and immich_postgresql
I can handle the fresh install, I could use help getting the backup into that though, never did that berfore.
first, lets locate the backups folder
(from immich container)
root@0cc3345e8164:/# ls /photos
backups encoded-video library profile thumbs upload
or from the root container just the dir you mounted
those processes are tied:)
oh, OK
so are you gonna wait until morning or start now?
I will run the paritycheck first
ok. Tomorrow I will not be able to help with the restoration process live, but will help offline. Before running parity check, copy out the dump somewhere safe
I am in no hurry, whatever works for you
looking for the dump now
just download those
-rw-r--r-- 1 nobody users 50815941 May 26 02:00 immich-db-backup-1748217600020.sql.gz
there is 3 from 24, 25 and 26 of may. I will get them all
only one? according to the date might be an already broken dump. No other dumps?
yeah, better get em all:)
Ok, thanks a bunch for your help so far, learning quite a bit here
I can help you restore just now in a voicechat and screenshare, this is gonna be quicker and easier for me
About how long do you think this would take?
guess about an hour. There is nothing to move or copy or wait for
hour max
OK, then I'm in 🙂
see you in a voice channel
umh, got to lokk for how that works
click the area below
you might want to close the issue (we solved it by restoring from backup)
For other folks tip to avoid is simple: do not use a share for pg, use an appdata. DEFINETLY do not use a share with mover
Thanks again for getting me back up and running!
Thanks for taking the time to help them out @pavelf! 🙂