Immich crashed for no reason
immich crashed for no reason.
I tried to add an external library and the server didnt start in 800s (normaly it starts in 500) and i reversed the change in docker.yml and the server came live after i dont know like 1200s and it shows running without problems, but the web and nothing works
How can i explain that in help desk
65 Replies
:wave: Hey @Maj,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Raspberry pi?
Noup.
Then keep posting all the things the bot asked for, sorry :)
Mac mini 2014, intel i5, 16gb ram, 128gb ssd, 1tb hd, running natively ubuntu
I tried everything from the list
there are no logs in docker
just shows as running
Also, i was running the 1.140 version
You haven't shared your compose file for instance 👀
You haven't shared exactly what you changed
Oh, sorry
There must be some startup logs
I forgot about that
I will do that
These are logs from immich_server
Docker-compose.yml
Under this:
- ${UPLOAD_LOCATION}:/data
- /etc/localtime:/etc/localtime:ro
I added:
something like home/home/DATA_HDD/2004-2023
Clearly "no logs in docker" lol
Is your database running?
Last time i checked there wasnt any. Maybe the computer freezed
ea8052d84362 ghcr.io/immich-app/immich-server:release "tini -- /bin/bash -…" 2 days ago Up 2 days (healthy) 0.0.0.0:2283->2283/tcp immich_server
24a094188a05 ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0 "/usr/local/bin/immi…" 2 days ago Up 2 days (unhealthy) 5432/tcp immich_postgres
2007c5451afb ghcr.io/immich-app/immich-machine-learning:release "tini -- python -m i…" 2 days ago Up 2 days (healthy) immich_machine_learning
5bb78f731e51 valkey/valkey:8-bookworm "docker-entrypoint.s…" 2 days ago Up 2 days (healthy) 6379/tcp immich_redis
Seems like it is
This is on web
It says the database is unhealthy though
Can you share the logs?
Your database might have corruption. Please follow the instructions in the FAQ
It’s one of the last ones
emm, i think this isnt ok:
home@home-Macmini:~/Documents/stacks/immich-app$ docker exec -it immich_postgres psql --dbname=postgres --username=postgres --command="SELECT datname, checksum_failures, checksum_last_failure FROM pg_stat_database WHERE datname IS NOT NULL"
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: Resource temporarily unavailable
Is the server running locally and accepting connections on that socket?
home@home-Macmini:~/Documents/stacks/immich-app$
Maybe try restarting the entire stack
home@home-Macmini:~/Documents/stacks/immich-app$ docker rm -f immich_postgres
Error response from daemon: cannot remove container "/immich_postgres": could not kill: tried to kill container, but did not receive an exit event
home@home-Macmini:~/Documents/stacks/immich-app$
I cant. I tried even restarting docker
You can reboot the host but my guess is the DB is cooked.
So complete restore?
But what caused it
If it’s corrupt, then yes you’d have to restore from backup
Usually bad storage device
Failing drive?
Databases are far less fault tolerant to small corruption than most other uses of a drive are, so it’s a common early failure sign
I mean i dont know, but i was transfering like 300gb of images from old and bad hd (external) to immich via immich go and it froze like 3 times completely
So your drive is failing under load, that sounds exactly right 😅
So that means probably the disk is cooked
Shit
Even if i had the db configuration on ssd?
Well the SSD might be what’s failing here
It’s whatever PG is stored on
But it’s hard to say without more logs or data
Is it possible that this happened because i was transfering data from (probably 1/2 dead) hdd
Yes, it could've been broken before
Well, did you restore from a database dump or did you copy over the postgres folder?
I will restore it now from database dump probably as this worked for me the last time
I meant when you were transferring data over from the dying hdd
Importing images from a failing disk wouldn’t corrupt a different Postgres disk, no
No. I have hdd internal in the computer with the immich images. Then i went and took an external disk and transfered the images to the internal hdd using immich go.
Then what Zeus says
The SSD postgres is on, is that an especially cheap one or something?
No, its apple quality.
Is it maybe that i tried to add external library to docker compose and it broke the postgres?
Well tbf that mac mini is 11 years old though 😅
Cause that was right after that
No that is unrelated
At least most likely
Yeah, i know😅
But im to cheap to replace it
But i didnt know it is corrupt
Ill try and restore the database dumb and ill post more details then
There's a chance it's not. As Zeus said it's impossible to tell from here
Will the immich freak out if the database has in it that some user has some pictures that arent in the upload folder
It will start up, the assets will simply show up as missing in the UI
How can i fix that then.
I mean i added one user for the images i wanted to upload and then deleted all of them as i wanted to add it as an external library
How would you end up with a database that lacks some assets?
i dont know
Deleted all of them through Immich or from the file system?
from immich
Why are you asking about this then?
Cause i uploaded it one day, then there was the db dump at 2am and then i changed the docker.yml and removed the assets.
I will restore the version before the change just to be sure
Ah got it
Yeah in that case you'll just gonna have to delete the assets from Immich again
from web?
You should definitely take backups in general though
Yes
Like database and upload?
Yes
But isnt the automatic database dump enough
Read the link
Emm, i need a little help for restoring:
home@home-Macmini:~/Documents/stacks/immich-app$ docker compose down
[+] Running 4/4
✔ Container immich_machine_learning Removed 0.1s
✔ Container immich_server Removed 0.1s
✘ Container immich_postgres Error while Stopping 24.1s
✔ Container immich_redis Removed 0.1s
Error response from daemon: cannot stop container: 24a094188a05454b8a592c74dbffa6efc3521a272096b72bfde8f0909e68608a: tried to kill container, but did not receive an exit event
I cant remove it
Try killing the docker VM
Or restart the host
it worked
I made the restore but now immich server is taking forever:
home@home-Macmini:~/Documents/stacks/immich-app$ docker compose up -d
[+] Running 3/4
✔ Container immich_postgres Running 0.0s
✔ Container immich_machine_learning Started 0.4s
✔ Container immich_redis Started 0.3s
â § Container immich_server Starting 873.7s
Now it worked after a really long time.