Server container crashing after miniature job request
Immich version 1.120.1 on windows running with docker desktop, was working fine for a month and more
I tried to rebuild the thumbnails with the appropriate job, the server container crashed. Since then it restarts and crashes again every few seconds.
The other containers start normally.
using
returns
69 Replies
:wave: Hey @Protenix,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.It's difficult to get the docker log of the exact moment it crashes, I've noticed two errors so far: Failed to decode DCT block (which I think is not relevant) and this
There's something wrong with your database. Probably some corruption in the filesystem, which is pretty bad. What do the postgres logs say?
just this
I'm not sure what to try next. Should I try to use the restore database function on this page? https://immich.app/docs/administration/backup-and-restore/
Should I update immich before doing this?
Should I do something different?
Future log output will appear in directory "log".The actual logs will be inside the postgres data folder
I'm not sure how to access them 🤔
The
postgres folder next to the compose file
Or wherever you set DB_DATA_LOCATION toin .env file DB_DATA_LOCATION=pgdata
However, there is no folder pgdata in the immich directory or subdirectory (and there never was). I think maybe it is on the linux vm or in the container files? I tried browsing a bit with the linux shell and with docker desktop "files" tab, but I could not find it
Thats a docker volume
Nevermind I found it in docker desktop container file browser under /var/lib/postgresql/data as stated in docker-compose.yml
This is the last log:
(It just repeats the last two lines)
This is a weird error
I went back to a log from yesterday afternoon
This is the first occurence of the error. Previous log was:
Your database is corrupted
You should restore from backup
Ok
Can i test my database backup to know if they are corrupted too (and which one) before restoring?
You would have to restore each one to test
Ok
I'll restore the db using the same immich version that I have already installed since it is the version that wrote the backup files, is it correct?
Yep
Ok. I'm not sure about an instruction in the restore procedure. This is the windows procedure according to docs:
I'm not sure on how should I do the instruction line
How should I edit my docker-compose.yml?
How do I mount the backup as volume in the immich_postgres container?
Youll have to mount that file under immich_postgres
Fair warning the restore on windows can be pretty challenging
I'm not sure how to do that in docker desktop
Is there anyone who has done this and could help me with this?
Not sure, in our docs we recommend using windows. I would recommend doing some google searches and read up on docker desktop
Basically you need to add that file as a mount next to the DB_DATA_LOCATION:/var/lib/postgresql line
I'm using docker desktop on windows
(The docker compose was the recommended installation on docs)
https://immich.app/docs/install/requirements/
"Non-Linux OSes tend to provide a poor Docker experience and are strongly discouraged. Our ability to assist with setup or troubleshooting on non-Linux OSes will be severely reduced. If you still want to try to use a non-Linux OS, you can set it up as follows:"
(ok sorry, you wrote "we recommend using windows")
Oh my bad i meant to write against. Sorry
Ok wait this makes sense, i've read it better
Ok I did as described in the restore procedure
However this error occurred:
I will try with the other backups
How did you make your backup? It doesn’t match the Postgres version
It is the automatic backup, but I started using the oldest one
Please post your docker compose
This same problem happens with the other backups
can you show me what folder you found the backup in, and what folders are next to that folder?
and you haven't edited your compose at all, or changes postgres versions?
hang on, that's not the file you are using ??
I:\backup\library\backups\backup1735092000006.sql is not the screenshot you showed meI haven't changed compose except to mount the backup, and I don't even know how to change postgres version
Youre not using the right backup
Please compare the path from the compose to the screenshot
the file i'm using for backup is the .sql file extracted from the .gz, renamed to avoid the "-" character
Ahhhhh ok. Can you show how you created it to confirm
I first copied the entire immich folder (including the filesystem) to be able to undo any changes. Then I opened any .gz file and extracted the .sql file inside (it has the same name as the .gz file). Finally I renamed the file to remove the "-" character and make testing easier. I did the same things with the other backup files.
You might have to edit the dump SQL file to remove LOCALE_PROVIDER and TRANSACTION_TIMEOUT
Do you mean to use a text editor to edit the sql file? The template1 problem would still remain, I think.
I'll try tomorrow
The template1 will be created if those earlier commands dont error out
Ok, I'll try
I might ask you to open an issue on github
Can you confirm youve never edited the docker compose since first setup? Until now
Can you post your env as well here
Yes
I never opened an issue on github but I will once the problem will be clear
I'll post my .env tomorrow, but i don't think It will help (iirc I just set the version of immich to use to avoid update problems)
In our first version of the automatic db dumps, we used pg_dump version 17 and trying to restore that into a pg 14 database causes these errors
You can try upping the postgres image to 16 (there is no 17 with vecto.rs 0.3.0), if that doesn't work then just deleting the broken lines from the SQL should get you there
I tried the second thing you suggested because Zeus suggested it yesterday
The backup restore went significantly ahead for a while, but in the end the main error remains
(log attached)
This is what I do:
Did you also try removing that LOCALE_PROVIDER option?
Yes
This line "Remove-Item -Recurse -Force pgdata" it's useless
If I understand what it is for, it should delete var/lib/pgsql/data which is already deleted by doing compose down, deleting postgres container
There is no "pgdata" in the immich folder
I'm sorry there was another "LOCALE_PROVIDER" I did not see
Just a sec
Ok it restored the backup correctly 🎉
Does this means that the database is healty?
If it restored and immich is working normally it’s likely everything is fine yep!
Can you try running this command?
docker exec -it immich_postgres pg_amcheck --username=postgres --heapallindexed --parent-check --rootdescend --progress --all --install-missingWhat will it do?
check for errors
I'm still figuring out which backup to restore, can I try it to determine the best (healty and more recent) backup or should I wait to run it after I choose it?
if it reports normal it means the backup is healthy, but you still have to restore it first
it's similar to the checksum failures you did, just an additional test
Ok I was wandering what --install-missing does since I don't want to modify immich installation on docker before choosing the best backup
it's a database extension, not the immich files
Running
944/944 relations (100%), 175546/175546 pages (100%)
This is all it says
that's normal
Ok perfect
I'll try to restore different backups starting from the most recent, the first one healty i find i'll keep it
How should I create the issue on github? Which problem do you want me to report?
Different db version between backup and live db?
this was fixed in 1.121 or 122
you are on 120, I didn't realize
either way if you delete those few words it won't affect your data. just annoying
Okok
I have a question: what about the files that are in the filesystem but there are not in the restored database? What will happen when I try to upload them?
you can use the repair page (immichurl.com/admin/repair) to find the untracked files, the manually re upload
you will have to carefully move them out of the immich library and reupload using the website
don't reupload the thumbnails just the original image
what happens if i don't remove them but just try to upload them?
not a big deal, you will just have some extra files in the library that aren't u sed by immich
Okok I was asking because I have some automatic backups on immich of some smartphones and they could reload them before I managed to clean the old ones
not a big deal