Docker has made immich unusable (changed the title to better reflect the problem)
i had a working immich install on my ubuntu 24.04 server, i came back home after a couple days and checked the dashboard and it shows this, if i use my existing email account it says "IMMICH SERVER ERROR"
any help appreciated, sorry in advance if i don't reply promptly.
164 Replies
:wave: Hey @Saladface35421,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:unable to get image 'ghcr.io/immich-app/immich-server:release': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
running the update command says this
$ sudo docker-compose logs
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
feels like docker is bunged up for some reason, i tried uninstalling and reinstalling docker
utk@grasshopper:~/salad1/immich-app$ docker compose up -d
unable to get image 'docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/images/docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8/json": dial unix /var/run/docker.sock: connect: permission denied
utk@grasshopper:~/salad1/immich-app$ sudo docker compose up -d
unable to get image 'ghcr.io/immich-app/immich-machine-learning:release-cuda': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
utk@grasshopper:~/salad1/immich-app$
and as expected backup commands aren't working either :/
Hmm, so Docker issue on your machine?
Maybe uninstall reinstall Docker will help?
Couldn't help but notice you using
docker-compose instead of docker composei tried reinstalling docker but the issue persists.
another "interesting" thing is i can't update/reinstall docker with
apt-get,
The repository 'https://download.docker.com/linux/ubuntu bullseye Release' does not have a Release file.that's because it's
stable not Releaseyep i caught that and fixed it, also using
/linux/debain was suggested somewhere, made that change and this error is now gone
but the main issue is still there
utk@grasshopper:~$ docker image ls
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? is the docker daemon running?
Does your user have permission for docker?
sudo service docker status
if not running
sudo service docker startdocker service is running, whenever I tried running
sudo dockerd i'd get an error that docker is running
I have stopped docker sudo systemctl stop docker and ran it again. currently sitting here wondering it is stuck or just taking it's time.
sudo dockerd
INFO[2024-12-10T06:58:15.706701646Z] Starting up
INFO[2024-12-10T06:58:15.707751953Z] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
INFO[2024-12-10T06:58:15.739779142Z] [graphdriver] using prior storage driver: overlay2
INFO[2024-12-10T06:58:15.742023254Z] Loading containers: start.
INFO[2024-12-10T06:58:16.203509183Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2024-12-10T06:58:16.339282523Z] Loading containers: done.
INFO[2024-12-10T06:58:16.350214902Z] Docker daemon commit=92a8393 containerd-snapshotter=false storage-driver=overlay2 version=27.4.0
INFO[2024-12-10T06:58:16.350285716Z] Daemon has completed initialization
INFO[2024-12-10T06:58:16.377662800Z] API listen on /var/run/docker.sock With sudo you get an error that it is running? What does that error look like?
failed to start daemon, ensure docker is not running or delete /var/run/docker.pid: process with PID 1189 is still running
i have a backup system image, i'm trying to boot into it. will update.
[+] Running 6/6
✔ Network immich_default Created 0.2s
✔ Volume "immich_model-cache" Created 0.0s
✔ Container immich_postgres Started 2.9s
✔ Container immich_redis Started 2.9s
✔ Container immich_machine_learning Created 2.9s
✔ Container immich_server Started 0.2s
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
during the new install this comes up, this is a completely fresh install of ubuntuDo you have gpu drivers for that nvidia card?
i was reinstalling nvidia container kit and getting this now
sudo nvidia-ctk runtime configure --runtime=docker
INFO[0000] Config file does not exist; using empty config
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.
utk@grasshopper:~/salad1/immich-app$ sudo systemctl restart docker
Failed to restart docker.service: Unit docker.service not found.
utk@grasshopper:~/salad1/immich-app$ sudo docker compose up -d
[+] Running 3/4
✔ Container immich_redis Running 0.0s
✔ Container immich_postgres Running 0.0s
✔ Container immich_server Running 0.0s
⠙ Container immich_machine_learning Starting 0.2s
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
major updatedocker is behaving now,
Fresh Ubuntu 24.04
Fresh immich docker pull
using old
.env & docker-compose.yml
install completed without any errors
webUI is accessible at port:2283
but can't access anything, attached screenshot
back to
docker compose logs we go thenon it.
could it be that the new install is not able to access the older database?
sudo docker logs immich_server
Error response from daemon: can not get logs from container which is dead or marked for removal
immich-app$ sudo docker compose up -d
[+] Running 2/2
✘ Container immich_postgres Error while Stopping 14.0s
✘ Container immich_redis Error while Stopping 14.0s
Error response from daemon: cannot stop container: ddeec11fa84a74b2bc049117684d1e2e90b3d64cc819d68e5c57dafdaadb9fe5: permission denied
truly astonishing, and i appreciate your patience through this allHow are you installing docker?
I'm surprised it's still messed up on a new install
i followed the installation guide https://docs.docker.com/engine/install/
Hmm that should be good
and it was a week ago, i left home while everything was working
had a long car ride yesterday, thought i'd backup on the way, didn't work, was confused, came back home to all this
Have you checked your disks' health?
yep, all good
plex is working fine
network shares are up
for years i have had the worst luck with docker, and it is back to haunt me
what happens to the data tho? is it just gone?
it is on a separate drive
images should just be in the folder
along with the database backups
what do you suggest i do now?
Seems strange that it stopped working during a backup
What exact commands did you run?
Also your Ubuntu server seems very cursed. Might be time to start over lol
i didn't run any cmds, i was just remote backing up photos from my phone to immich
and i did do fresh install of ubuntu a few hours ago, i don't mind going through all that again as long as i can access my photos 😢
Is the database also on the network?
this is my setup:
i5-2400
gtx1660ti
Ubuntu 24.04.01
Boot SSD
1TB HDD for photos
2TB HDD for other linux isos
1tb hdd contains two folders
immich-app and Photo
Immich-app contains the docker-compose file/ .env
Photos contains another folder utk |
.env file
# The location where your uploaded files are stored
UPLOAD_LOCATION=/home/utk/salad1/Photos/utk
# The location where your database files are stored
DB_DATA_LOCATION=/home/utk/salad1/Photos/utkBruh
You’re using the same folder for the immich files and the database?
👀
We have got to stop helping people who don’t post their compose and env
No more
This is a massive conversation that’s almost certainly due to absolutely busted permissions on your filesystem since you have two services expecting exclusive lock on a folder
I’m sorry to vent but it’s frustrating
i'm really sorry guys
but
...
The good news is this is probably fixable if you move the folders apart
...probably
There’s a few issues here. First of all Postgres has probably changed the permissions on the entire folder to make it non writable to immich
My guess is in the process of trying to fix that your docker installation got broken from the reinstall attempt
Ok so move the upload, encoded-video, thumbs, profile, library folders somewhere else, like
/home/utk/salad1/Photos/utk/Immich or whatever
that'll be your UPLOAD_LOCATION
Then move everything else to /home/utk/salad1/Photos/utk/postgres
That'll be DB_DATA_LOCATION(And backups)
Ah I knew I forgot one
For reference, a normal postgres folder looks a little something like this
Now I know your hostname brb hacking into the mainframe
Joke's on you, I have no frames
why zeroTrust when you have ZeroFPS
If you've moved it all, do
sudo chown -R 999:999 /home/utk/salad1/Photos/utk/postgres and sudo chown -R utk:utk /home/utk/salad1/Photos/utk/Immich (paths of course as example)just a minute
quite a bit of data, taking a while to copy
ok i have updated the permissions
what is the user 999? should it be my user "utk"
In the case of postgres files, I very specifically said 999 and not utk
for some reason when i do that this happens, thus the confusion. screenshot attached
that's fine
try
id plex it should output uid=999yep it does
ah the
machine-learning folder is something not from postgres 👀 ...I think
What's inside?machine-learning>models>...nothing...
curious
just leave it be then
Now there's a good chance this all failed and we're going to have restore the DB from backup, but that's ok because you have those, right?
yep
Because postgres is kind of fussy about being used like this
also, i can see the photos in front of me
let me backup them once, just in case
thanks for waiting, i have manually saved all the photos on a separate system. all set to go forward
lmk what i should try next
If you did those two, just start immich with the new folders in the env var
i think the containers are bunged as well, let me try and i'll follow up
okay so we are back to where we started
utk@grasshopper:~/salad1/immich-app$ sudo dockerd
INFO[2024-12-10T14:55:19.436614851Z] Starting up
failed to start daemon, ensure docker is not running or delete /var/run/docker.pid: process with PID 37045 is still running
utk@grasshopper:~/salad1/immich-app$ sudo docker compose pull && sudo docker compose up -d
unable to get image 'ghcr.io/immich-app/immich-machine-learning:release-cuda': Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
utk@grasshopper:~/salad1/immich-app$
last time i tried to delete /var/run/docker.pid it just came up with a new errordon't delete the pid, is the process running?
sudo ps aux | grep container
or is it containerd 🤔
that should worki didn't delete
/immich-app$ ps aux | grep docker
root 37045 0.6 1.0 2800452 88960 ? Ssl 14:52 0:01 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 40832 0.0 0.0 1745288 4224 ? Sl 14:57 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 2284 -container-ip 172.19.0.3 -container-port 2284
root 40839 0.0 0.0 1819020 4224 ? Sl 14:57 0:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 2284 -container-ip 172.19.0.3 -container-port 2284
utk 40981 0.0 0.0 6544 2304 pts/2 S+ 14:57 0:00 grep --color=auto docker look at that
sudo service containerd status ?wait i didn't notice the edit
active (running)
● containerd.service - containerd container runtime
Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-12-10 12:08:32 UTC; 2h 51min ago
Docs: https://containerd.io
Main PID: 895 (containerd)
ok so first off, stop doing
dockerd and only do the service start/stop
it's auto-enabled anyway so it should autostartyou mean the
containerd.service, right?they're 2 services
dockerd uses containerd
now do now do
sudo usermod -aG docker utk
next up exec bash
now id utk should list dockeryep it does
and then you
sudo service dockerd restartutk@grasshopper:~/salad1/immich-app$ sudo service dockerd restart
Failed to restart dockerd.service: Unit dockerd.service not found.er, it's without the d
got it, done
Now what does
docker ps sayutk@grasshopper:~/salad1/immich-app$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?hum hum hum
sudo ls -l /var/run/docker.sockNo such file or directory
I see 👀
and the docker service is running?
yep
could you
sudo ls -la /var
alright that's good, and
sudo ls -la /run ?everything belongs to
root:root excpet for a couple lxd and utmpno no post the screenshot hombre
this is important
may i ask what are we checking here?
goddammit snap
ownership of the docker pid/sock files
got it, and what makes
snap a b.....sudo systemctl status snap.docker.dockerd.serviceUnit snap.docker.dockerd.service could not be found.
pfew, ok
Fucking snap lol
docker --version is 27?You linked the page you used earlier, but can you detail exactly how you installed docker?
Me @ your Ubuntu installation at this point tbh
I don't really understand how the docker service can be running without a
/run/docker.sock😭
It's literally fresh`
Huh
Oh wait different topic
Too much support today
27.4.0
Alright digging deeper
sudo cat /lib/systemd/system/docker.service | grep ExecStartExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ok so it's using the containerd.sock, could you
ls -la /run/containerd/ ?literally the boot USB i used to reinstall ubuntu earlier today, ignore the ethernet cable, i ain't compensating for anything
Good ol' SanDisk 16/32GB
This please
i mean i just followed the instructions to install with apt
We need much more specificity than that
Links and exact commands run
Also what exact Ubuntu server image have you installed
i can just copy paste the commands here as i did while installing
inb4 it's fixed with a
sudo service containerd restart && sudo service docker restartfor pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
did ithttps://superuser.com/questions/1741326/how-to-connect-to-docker-daemon-if-unix-var-run-docker-sock-is-not-available this thread also details how you can get the error even when doing everything right
Super User
How to connect to Docker daemon if unix:///var/run/docker.sock is n...
I had an issue with docker installed with snap, and I moved to apt. This was on a production server. In order to have a low downtime, I did the following:
Removed snap running containers
Removed /...
(And some fixes)
Personally I think https://superuser.com/a/1852442 this answer is the best if that fixes it
utk@grasshopper:~/salad1/immich-app$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
check this
did it work?
dinner time, be back in a while
btw the webUI is not accessible
oh noooo
Yes
docker compose logsThat looks pretty OK to me 👀
How are you trying to access the UI?
dOn'T heCk my IP lol
I'm in :HACKERMANS:
It’s coming from.. INSIDE THE LAN
mmm and if you just
curl http://localhost:2283 on the server?curl: (7) Failed to connect to localhost port 2283: Connection refusedDid you post your compose here somewhere?
What if you just reboot the server at this point 👀
i did that, containers are healthy but still no web UI
immich_server isn't healthy anymore
Logs and
docker inspect immich_server pleaseso there was a sudden power loss and my server restarted, webUI is now up
your server is cursed 100%
wtf is happening
this is where i am right now
can't create the admin account, which is odd imo because it shouldn't be asking me to make one anyway, i have used immich...
I think your splitting of the DB_DATA probably broke the database
If docker is working you should probably start with a brand new immich install
at this point i am too afraid, you mind guiding through it?
It would be a fully standard install. Just pick a. New folder and use our guide
https://immich.app/docs/install/docker-compose/
Don’t edit any paths in the env or compose at all. Just make a new directory in your home folder and do it in there
how about i delete the containers and the docker image, delete the data in the
postgres folder, keep the data inside the immich folder and docker compose up -d again?
with .env looking like this
# The location where your uploaded files are stored
UPLOAD_LOCATION=/home/utk/salad1/Photos/utk/immich
# The location where your database files are stored
DB_DATA_LOCATION=/home/utk/salad1/Photos/utk/postgresNope
You can’t use the upload_location without a matching database
got it, so completely wipe
immich folder as well, right?You can keep it to the side if you have pictures you want to save
But you should use a brand new folder for this install
Also what is /salad1
Can you run
df -h?salad1 is the folder where my HDD is mount, it's not stupid if it works, right?
You should avoid using any mounts for now with how messed up your system is
And you should not use a mount for Postgres at all
I’m sure there’s an element of external drives not being mounted here. That could explain why it gets reset
so keep the postgres folder somewhere on my boot drive? any suggestions where?
i'll need the mounted hdd for the photos tho
Doesn’t matter where. I guess mount for the photos is fine
update: /home/utk/immich_postgress will be the
db_data_location (this is on the boot drive)
/home/utk/salad1/Photos/immich will be the upload_location (hdd is mounted on salad1)
since i have complete backup of the upload_location folder (that means all the encoded videos and the photo...) how about i just delete the entire folder and make a new one as mentioned above.
step 1: let immich install
step 2.A: manually copy paste the library and the encoded videos and nothing else and restart immich?
step 2.B: upload everything again using the webUI
step 3: pray?Why would you copy paste the library and encoded video?
I feel like you don’t understand how the upload_location works
you should not ever move any files into or out of this folder
got it
thanks, i shall update you after immich is installed
1. Install immich into brand new folders
2. Upload photos to immich using web or immich-CLI or immich-go
got it
confirming the permissions, the immich_postgres is owned by root:root and immich (upload_location) is owned by utk:utk
should i make any changes before installing?
Why is immich upload location not owned by root?
You shouldn’t even make the folders to begin with, let docker make the folders
You’re like adding all kinds of extra steps that aren’t in our docs
It makes it very hard to help
my bad, sorry and thanks
guys
it is working
thank you all soo fucking much
Wow 😱
🤣
once it is done uploading i'll get to github and fugure out hw_encoding and ML
i have a gtx1660ti 6gb, should be plenty good
and will take some stress away from my poor old i5-2400
Make sure you set up solid backups as well
i am thinking that i should change the default backup location to a separate drive, just in case this drive fails or is corrupted the entire backup will be on a separate drive
Wdym default backup location
You should be making separate drive and offsite backups of all your data
For the DB dumps
yep yep
also is there an automated way i can backup the photos to a separate location? or any suggestions about what you use personally?
Restic is nice
Many people also use borg
I think you should just setup backups and not muck around with the UPLOAD_LOCATION mounts based on how things have gone so far haha.
If you want the database dumps on another disk you can use rsync every night cron job
that's true, that's why i was hesitant on doing anything with my install. didn't even mess around with hw_transcode and hw_ML either, it just works so i didn't bother :peepoCryingAlot:
how do you suggest i setup a periodical backup of the data and which folders should i backup
You can use restic - backup all folders in UPLOAD_LOCATION except for thumbs and encoded-video
i'll look into it, thanks
also, thank you all for being super awesome and helping me out :widepeepoHappy:
and i still don't know how a power outage fixed my docker, but i ain't complaining