Setting up immich for first time and the immich_server container freezes my entire VM
I'm a noob, but I'm pretty sure I'm not doing anything wrong lol
I'm using portainer, following the instructions on the website. I have the latest version of docker engine installed on a Debian VM in TrueNAS.
https://immich.app/docs/install/portainer
I believe the immich_server container fails to log into some service or services (see log), and that it retries infinitely, fully freezing the VM. The only way I fixed the frozen VM was restarting the VM and manually stopping the various immich containers until I figured out which one was causing it. I still don't know why it was causing it.
I haven't been able to get immich to launch, I can't get past starting the immich_server container. The other containers (machine_learning, postgres, and redis) run as "healthy" according to portainer. The initialization seems to go well until immich_server attempts to log into something, then I get hit with
[Nest] 17 - 01/12/2025, 3:32:38 PM ERROR [TypeOrmModule] Unable to connect to the database. Retrying (7)... error: password authentication failed for user "postgres"How do I fix this? Attached are my "compose" file, my .env file, and I attached a log file showing two attempts to start the container. You can see in the compose file: * I changed the .env to stack.env. * I set restart to "no" for immich_server because the only way I can stop it from locking up my VM entirely is to kill the service through the cli as the VM starts up and before immich_server starts attempting to log into the database. Setting it to not autorestart makes this testing a little easier, and is not how I initially had it configured- it's not causing problems. * I set the paths like /dockermount/immich/database, which is a directory on my TrueNAS machine that I mounted in the VM through NFS. It's how I do my other containers (like NextCloud) and it works fine. I don't think it's a problem.
56 Replies
:wave: Hey @fekenator,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJAChecklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.GitHub
immich-app immich · Discussions
Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.
FAQ | Immich
User
GitHub
Issues · immich-app/immich
High performance self-hosted photo and video management solution. - Issues · immich-app/immich
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
You should never use NFS for a database. Local storage only
fwiw, it's on the same machine, the VM connected to TrueNAS through a network bridge
Yes, doesn’t make a difference here
Databases need close contact to the storage layer for sync writes etc that isn’t as reliable over NFS
is it even possible that that's the cause of my problem?
I don't think a poor database sync would cause an incorrect password issue?
Given that there’s an issue connecting to Postgres is seems plausible. I don’t see the logs
Are these the relevant logs?
I assume the machine learning one isn't lol
(but I am a noob!)
did you not rename the example.env to .env?
I'm using portainer
so everything is just copied in
the compose file is exactly what I pasted into the editor, and the .env file is exactly what I pasted into the advanced view environmental variable section. I did set the timezone in the environmental variables, but only on the like 3rd try
you can check the postgres logs in the logs/ folder of postgres for more logs
it's most likely that you either changed one of the DB_ variables or started it once before setting them and they're now the default
I nuked the volume folders between each attempt, and didn't start anything until I had set every variable and made every change
I deleted the stack, and the containers between each attempt
if you want me to, I will do it again
Did you delete the postgres data volume as well?
I'm not sure. I think so?
So, I mapped two volumes externally- library and databse. I deleted then remade both those folders between each reinstall
everything else was deleting the containers through portainer
and then the stack itself
I don't know if there's some other sneaky place database files could be hiding, but these measures have worked with other services
I'm not sure what's going on, but most likely is some issue related to 1) env vars, 2) DNS/host resolution, or 3) using an unsupported mount for the database
the password authentication error is not always truly indicative that everything is working fine other than the password
the locking up thing is likely an issue with your hardware and/or hypervisor
You could try connecting to the database using
psql and the expected username/pw and see what happens, and investigate the logs folder for more details as wellif you check the logs, I think it's just immich_server flooding login attempts that makes it freeze. When I set it to not restart, it stops after 10 tries, but an earlier log had.... lots
sweet, I'll look into psql
what resources does the VM have?
at max, 4 cores 8 threads, 32GB RAM
this is running on a xeon-2125, this has happened whether I give the virtual CPU 1 core or all 4 cores, and between TrueNAS restarts
my first attempt, I gave the VM 10GB of RAM, and TrueNAS was showing about 8GB utilized. I changed it to 32GB before the next attempt, and usage went to 12GB, then 16
and seemed like it would keep going
I think it's very unlikely that immich can lock up an entire system with password attempts, we see people with this password issue from time to time and never seen that before
so I stand by hardware and/or hypervisor for that
fair enough
maybe your NFS is going crazy and killing the network traffic or something like that preventing access to the VM
just a random thought
so what would a supported mount even be? just make a folder inside the VM and try again?
oh yeah nvm lol
I'll give er a go
Yep. The image storage is fine for NFS but not the database
this is off-topic then, but how does that reconcile with....
or like, I've heard a lot how people will host VMs on one machine, but keep all the storage on a different machine, especially for enterprise or business stuff
is it just implied the databases are still residing on the VM machines?
if it's too off-topic to get into that's fine!
Typically the database will be running on a dedicated server and then connect to the apps over the LAN
There are some enterprise storage options that can be reliable for databases but not ones the average user typically uses
The main one is iSCSI
https://www.postgresql.org/docs/current/creating-cluster.html#CREATING-CLUSTER-MOUNT-POINTS
PostgreSQL Documentation
18.2. Creating a Database Cluster
18.2. Creating a Database Cluster # 18.2.1. Use of Secondary File Systems 18.2.2. File Systems Before you can do anything, you …
ahhh gotcha
so if I were running this as a block device over iscsi it might be more copacetic?
Maybe but if youre doing that you are well beyond our help tickets
I haven't used iscsi yet, but I will be soon for a couple things like lancache. There's lots I won't know about it but 🤷♂️
We will help you with the standard setup, and some deviations that are known to work
ha, definitely
Past that you are on your own lol
for sure for sure lol
bad news though, I just tried it locally
no dice
CPU to 100, all other containers are unreachable
this might be a dumb question
well no, it is dumb, right- should I just let it sit for an hour running flat out? see if it just needs time lol?
but that wouldn't make sense if it's failing to log in
No, that shouldn’t be needed
well, tried again with docker compose locally on the VM
no dice
Is this the kind of issue that, if it were an issue with the actual images rather than just my specific system, it would be widely known by now?
100%
well that's good to know at least, in its own way!
Thanks for the help! I think I'll give up lmao and if I reinstall this VM ever, try again
why not try with a brand new VM?
make a new ubuntu VM, install docker ( https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository ), then use our install script ( https://immich.app/docs/install/script/ )
there's like no way that won't work
then you can move files and stuff once it's at least working in principle
because I'm burnt out :p but if you insist!
I did it again through proper docker compose on another debian vm I made a couple weeks ago and have barely touched
it's a very different looking process now that I hit docker compuse up -d
It's spending a lot of time pulling things for each of the containers
well fuck me, it's working
yay!
the pull is probably because portainer had already pulled the images on the other system
or at least, I can access the initial landing page
it's working
already pulled the images
now you could shut it down, change the UPLOAD_LOCATIOn to an external drive, reset the database folder and start again
like it had basically cached one or some of them?
docker will cache the images once first pulled yes
but not related to this issue
sorry it took me a good 5 minutes to get what you meant, I've been up for 20 hours lol
so like, shut down the containers, edit the .whatever file to change the upload location somewhere networked as a test, and then.... reset the database folder?
Reset it as in rm -f it and remake it, then start the containers again?
and yeah thanks even more man, this is at least proof it works- and on a vm from the same ISO as my main one. So now I feel a bit committed to making that work, but I probably won
't get far until tomorrow
if you do
docker compose down, change the UPLOAD_LOCATION to point to NFS, rm -r the postgres folder, and start it back up, you will get a fresh start on the NAS
alternatively you could move the current library folder to NAS but if empty, no real reason tofantastic, I have a clear course of action then. I'll start tomorrow's tinkering with that. I'd do it now but it's like a 5 step process to fumble my way through lmao, before I even touch docker
I guess a last question- do you think there's any chance I get it working on my first VM?
I can't tell anything that would be different between the two, other than the second VM pulling a lot of new files
you were using portainer right? so you didn't see the pull on the first system
it still happened in the background
there's no reason it shouldn't work on the first VM but immich usually just works, so IDK what could be going wrong. I would put my money on portainer env var issues
oh duh, sure
I still tried it a couple times outside portainer though, through docker compose- the exact same way I did it on the second VM- so it just doesn't feel like it would be portainer, unless it messed something up from the very beginning that docker compose then used or was affected by. I made brand-new directories in a different location and everything
I always planned for these VMs to be able to be nuked while easily saving all my data, so hey maybe this will just turn into an earlier lesson on that than I intended!
Did you use our script or just use the compose and env yourself?