mmich_postgres container crashes with “permission denied” on /etc/postgresql/postgresql.conf after u
SOLVED!
https://discord.com/channels/979116623879368755/1376486772900499487/1376899119364178112
I’ve followed the Breaking Changes guide for Immich v1.133.0, updated my Compose file as documented, and switched the database image to ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0. My Compose snippet for the database service is:
When I run:
docker compose down && docker compose pull && docker compose up -d && docker compose logs -f
the immich_postgres container repeatedly exits with:
LOG: could not open configuration file "/etc/postgresql/postgresql.conf": Permission denied
FATAL: configuration file "/etc/postgresql/postgresql.conf" contains errors
and then the Immich server fails to resolve the database host:
Error: getaddrinfo ENOTFOUND database
What I’ve tried so far:
Verified file ownership & permissions on the host under /var/lib/docker/bindmounts/immich/postgres:
-rw-------+ 1 lxd docker 29K Jun 2 2024 postgresql.conf
Commented out the custom healthcheck and command: sections, as per the Breaking Changes.
Ensured the .env settings point to the correct DB_DATA_LOCATION.
Expected behavior:
The Postgres container starts normally, migrates the database (VectorChord), and Immich server connects to it.
Actual behavior:
Postgres crashes with a permission error on its own postgresql.conf, preventing the server from ever resolving the database host.67 Replies
:wave: Hey @palpatine,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:I was trying pulling last image but get the same issue:
Hmmm, this is the second report we have seen of this, you took these permissions from the host but this folder isn't bind mounted
-rw-------+ 1 lxd docker 29K Jun 2 2024 postgresql.conf
I am confused where you're pulling this from in this instance
Do the permissions look the same for this within the container?This is what I expect to see in
/etc/postgresql when running the container as root
I'm unsure why some people are getting different permissions there
Maybe it's an LXC thing?
Im confused too, i was undo the changes
and start the container. See log file and immich is running
Undo what changes?
original compose file with
Ah, yes version 1.133.1 can run with the old image, but that doesn't really solve the problem
We need to figure out why some people are having these permissions issues
ahhh okay i understood okay if i switch back i get the same issue
currently i have this permissions:
for
/var/lib/docker/bindmounts/immich/postgresI see, that's not the location of the file having the error though
The file that is having the problem exists only within the container at /etc/postgresql
Can you add
command: tail -f /dev/null
to your compose, this should stop it trying to boot postgres and allow you to exec into the containeris that useful inside container ?
Can you go into /etc/postgresql and look at the permissions of the files in there?
Oh, you did
There's no files? 🤔
Just to confirm, this was 100% the new container?
Because it should be impossible for the /etc/postgresql folder to be empty
oh no sorry! one sec
Ah good :KEKW:
noits the old image. I switch back
If you could switch to the new image again, and then try to exec in, as I said before you may need to add
command: tail -f /dev/null
to your postgres docker compose section as the container will probably automatically shutdown after it fails to read the config
So you won't be able to exec ini start new with:
docker compose logs -f
docker exec -it immich_postgres ls -l /etc/postgresql/postgresql.conf
-rw-r-----+ 1 root root 470 May 26 09:46 /etc/postgresql/postgresql.conf
docker exec -it immich_postgres cat /etc/postgresql/postgresql.conf
Can you run this in the container?
grep UMASK /etc/login.defs
Can I get both of
docker exec -it immich_postgres ls -l /etc/postgresql
and
docker exec -it immich_postgres grep UMASK /etc/login.defswith
command: grep UMASK /etc/login.defs; tail -f /dev/nullAh, I didn't mean put it in the command second 🤣
You only need the tail -f /dev/null in there so that you can exec into the container 🙂
It did still show me what I needed though, which looks fine, making it even more confusing
Could you run this one for me?
docker exec -it immich_postgres ls -l /etc/postgresqlYes I need a bit. I need to handle my daughter currently
No worries 🙂
If I can help with any Future information let me know. I think if it a big problem I could thinking about a remote session if it is useful 👍
That may be helpful, I have a fix that should work but I really am trying to understand the root cause of this before just applying a fix, don't want it to bite us again in future
Can you check where your data root lives? It might show you in this file:
/etc/docker/daemon.jsonGood morning
Yes, I will check it now
btw. nice tip
command: tail -f /dev/null to hold container 🙂Ok so your data should be in /var/lib/docker then
So could you run
getfacl /var/lib/docker and let me know the output?So I believe the other::--- means that it removes all permissions from the other bit by default
If you're willing, you could try the following
This should change the other section to allow read by default, alternatively you can remove the other override entirely which will mean whatever is set for other won't be overridden by the ACL
Same
2025-05-27 08:55:20.401 GMT [1] LOG: could not open configuration file "/etc/postgresql/postgresql.conf": Permission denied
my .env:
This one is not working too:
sudo setfacl -m default:other::r-- /var/lib/docker/bindmounts/immich/postgresCan you confirm it actually changed the ACL correctly by running getfacl again?
yes:
sorry @Zack notifications are not working currently :/
Hmmm, I see other:: is still there
Maybe you need to change that one too
Let me grab the command for removing an ACL
Lets try running these two
sudo setfacl -x default:other /var/lib/docker
I'm trying to get my head around reading these ACLs, I haven't got any experience with them myselfI will try it.
same
I'm currently trying to spin up a debian VM so I can test this out myself
So, spun up my own debian 12 VM with docker and I can't reproduce this problem
Could you run
getfacl /var/lib/docker/overlay2 please?Yes sure
I'm really struggling to reproduce this 😅
I don't know the reason bro sorry 😅
Maybe it's a Ubuntu cis server level 1 problem ? But my understanding is that this kind of issue should be there in past not only for this new container
in the overlay2 directory, can you run
find -name postgresql.conf -exec ls -la {} \;Hmmm, so your overlay folders are getting the ACLs applied to them, mine are not
Curious
maybe ?
cis_level1_server
Alright, I still personally can't reproduce it because I can't get the merged overlayfs to inherit the ACLs
However, we can try this on your system
setfacl -m default:other::r-- /var/lib/docker/overlay2
setfacl -m default:mask::rwx /var/lib/docker/overlay2:monkaHmm:
So your ACLs for your overlays must be coming from somewhere else...
I think I might've figured it out
REPRODUCED IT
WOHA!
Ok ok
So, my changes to overlay2 are correct
The problem and why it didn't fix it for you is that overlayfs is merging two different folders
The base image, that already got pulled, and the changes made in the container
The ACL for the base image is set when the image is pulled, so any changes we made to ACL weren't applying
It seems overlayfs takes the ACL from the image, rather than the new layer
So to fix this now your ACL is correct, we need to remove the old image, and re-pull it
So, if you docker compose down your immich stack
Then
docker image prune --all
Then docker compose up the immich stack
I think the new ACL changes will get applied, and your postgres will start 👀sounds logical....a little bit 😄
there where some old images ....
Total reclaimed space: 38.23GB
Wew
seems running
Thank the lords
yeps its up
awwsome!
Awesome!
awsome
Now I will apply the fix to our postgres image so this isn't a problem in future :KEKW:
Thank you very much! You're much better than chatgpt 😄
Thanks so much for your help running commands and such on your system 🙂
Haha, I actually was getting some assistance from Gemini, it's a good brainstorm partner 😛
Nono, i need to say Thank oyu! You and all of immichs team doing great work!
Have a great rest of your day, we'll have a new image out soon to fix this problem, if you want to revert your ACLs to disallow other again then, that should be fine 🙂
What a wild ride 👀
u2 🙂