Crash on "Generate Thumbnails" - Missing/Corrupted files
I had a few images with failed thumbnails, but every time I try to run it again the container crashes.
Running on docker container, .env :
IMMICH_LOCATION=/home/serv/disco2/backup/Quelass
DB_DATA_LOCATION=/immich/postgres
CACHE_LOCATION=/immich/cache
THUMB_LOCATION=/immich/cache/thumbs
ENCODED_VIDEO_LOCATION=/immich/cache/encoded-video
BACKUPS_LOCATION=/immich/backups
TZ=Europe/Lisbon
IMMICH_VERSION=release
Mainly 2 dirs for cache and db in ssd and main storage for upload in hdd.
Error at the time of crash can be found in the attached txt.
------
Not sure if related, but also got some missing thumbs when just opening the homepage on one of the accounts - missing_thumbnails.txt
Appreciate any help/tips, I tried to look around for solutions but found that missing files/offline paths seem to be a nightmare to fix, so came here to get some help 😄
45 Replies
:wave: Hey @Golfinho,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:Woke up today to even more good stuff...
Got thousads of erros liek this:
all for the user 85.... on the folder thumbs.
Also did a little bit more diggin and the uplaod fodler and thumbs has 1 less user than uploads:
root@e75051c9a764:/data/thumbs# ls
4d0bd5f7-83e7-4ffe-ab94-d6637e2da04b 85e803c7-77c7-4048-9487-012c1972c920 d33c27ed-a95f-434e-a3e2-117f571682c9 f51c90b6-9a56-47ee-8fdc-190f735cdfd5
65e8da94-ff73-4d46-8694-eecaf852a566 b23def0e-2bdb-499b-b9dc-c4b72e1c21e9 efb632b5-0498-451c-8a93-44c3d5710114
root@e75051c9a764:/data/thumbs# cd ../upload/
root@e75051c9a764:/data/upload# ls
4d0bd5f7-83e7-4ffe-ab94-d6637e2da04b 80424f7b-9998-4ada-917e-66c8621c8b4a b23def0e-2bdb-499b-b9dc-c4b72e1c21e9 efb632b5-0498-451c-8a93-44c3d5710114
65e8da94-ff73-4d46-8694-eecaf852a566 85e803c7-77c7-4048-9487-012c1972c920 d33c27ed-a95f-434e-a3e2-117f571682c9 f51c90b6-9a56-47ee-8fdc-190f735cdfd5
And the user that is missing is not the one of the errors, the one from the erros seems to be both on the uplaods folder and on the thumbs folder
Are these folders mounted on a network storage @Golfinho ?
Nope, everything local
The / is mounted on an SSD, /home/serv/disco2 is an internal SATA HDD
This first SSD, is that USB?
Sorry for the late reply, but both are internal
the first ssd is an nvme connected by pci3 x1, the hdd is a sata3 portinternal.
Both have been on the same setup for a while now, never gave me any trouble
The only thing that seems concerning to me is:
You're mounting things inside the ML cache folder that aren't supposed to be there, which may have undesired side-effects
How many assets do you have @Golfinho ? Because it may be easiest to just re-generate all thumbs
I'm not really sure at this point, but I have an idea the cache folder already holded the encoded-vids and thumbnails? maybe I confused cache location for the immich server vs my understanding of cache while separating the ssd and hdd? should I replace cache_location for a new separate folder?
this is my cache folder
Yeah the cache location you use is ML download cache
You could move it to a subfolder for instance
But I think the only way to fix the missing thumbs is to regenerate all of them 👀
What would be the procedure to do that?
Shut immich down, move your folders to for instance:
And then start back up
If that all starts up fine, you trigger the "ALL" button on the thumbs job
https://my.immich.app/admin/jobs-status
Moved the thumbs and encoded vid into a backup dir, created the folders outside the cache dir, added .immich file, gave permissions and regenerated all the thumbnails, created new thumbnails for most of the photos, but still there are some that when it tries to generate crash the server, it reboots twice and then boots up again
When it reboots it shows 0 missing on the thumbnails job, bit clicking on "missing" on that job the server instantly crashes, reboots twice and then it's up again like nothing happened
I'd really like to keep the current album structure and faces identified, but I don't mind having to regenerate everything else if that could help, I don't mind if it takes a long time
Anything in the logs that identifies the specific asset it crashes on?
Because I imagine one of your assets is simply corrupted
Or multiple
when i have docker logs -f on immich_server and click on generate thumbnail it instantly crashes, not even erro msg in that container
will try to check logs on the other containers
and it says 243 failed on the thumbnail job
To identify offending image(s) I suggest to apply a one-line monkey patch to enable logging each thumbnail generation attempt. By default they are not logged.
When Immich crashes, check log lines right before the crash: the offending asset will be among last lines. (Depending on concurrency settings, it may be the latest
[trace] Generate thumbnail or earlier)
To confirm your finding: navigate to the photo in timeline or viewer and trigger "Refresh thumbnail" from the three-dot menu. Does it crash the server?
(This is based on https://github.com/immich-app/immich/issues/21883#issuecomment-3312059270)logs start on the moment i hit the generate missing thumbnails job
and end with the server crash
the file mentioned in LOG [Microservices:MediaService] [trace], as far as i can tell, seems fine, can open it on windows accessing the uplaod dir, has metadata and doesnt seem to have anything wrong 🤷♂️
can you zip the file and share it here?
oh wait, i think i might have something
does immich support .svg?
i stopped being dumb and instead of 16 simultaneous thumbnails changed it back to 1, and the output from pressing the job button to the crash not is:
and the last thumbnail is a .svg
a diffrent extension, probrably one not expected, that must be it no?
It should be, yes
if i were to change it to a .jpg through a converter, coul i swap the files directly? (same name,etc, just diffrent extension)
or would that cause caos in the db?
That would cause chaos, but we still have a "replace with upload" button in the GUI
SVG?!
This is amazing.
The issue I linked above https://github.com/immich-app/immich/issues/21883 where SVG causes crash. That issue is virtually non-reproducible because it is very specific to environment.
Could you tell more about your hardware, OS, setup?
Sure!
immich is running on a i3-12100, 16gm so-dim ddr4 ram, ubuntu, running inside the docker image, should be running the latest version.
uname -a:
Linux serv 6.14.0-35-generic #35~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Oct 14 13:55:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
SVG is attatched
will give this a try later, but now i'm down to help troubleshoot this issue
doubt these matter, but:
running immich behind nginx with fail2ban setup, part is on ssd, part is on hdd:
Everything inside /disco2/ dir is on a spining hdd, the rest is on an nvme ssd
tried to swap it for the jpg converted image of itself with the "replace with upload" and the server instantly crashed 🫂
tried again the generate thumbnails, and got another crash on .svg
i could swap the first .svg, but this one when i search for the name of the file or extension, when i try both "close8.svg" and "04da3633-3c74-441e-810d-2b6c54b7cbe8.svg" the web ui doesnt find any image (it is in that directory, i checked it manually)
Could you also collect debugging output from ld (linker) using the commands below and post 3 generated files?
It runs sharp in containers, attempts resizing a minimal svg, saves linker output to files.
- Immich: default container
- Immich: with empty LD_LIBRARY_PATH
- NodeJS
nvm im dumb, its in another's user space, so i have to switch to find and swap it
i logged in with the user of these .svg, found 2 more, but both seem like ghost files for immich, i cant delete them, it doesnt show a preview, doesnt have hte full options on the top of the image...
"error loading image" in the middle, i dont have the option to delte, swap or do much with them
wait, the same file is duped on another user
the partner thing
That was it, deleted every .svg from all users and now the server doesnt crash on thumbnail generation!
i still get a lot of erros tho, not sure if I should be worried about those:
and many many more
There are some files that simply do not process well, a couple of dozen/hundred is normal for 100k libraries
Thank you for the ld debug logs. Option 1 default Immich - crashes; two other options - work fine and print
resize OK.
Regarding other issues/logs:
VipsJpeg: premature end of JPEG imageYou can try force processing those not-100%-well-formed images by setting env var
IMMICH_PROCESS_INVALID_IMAGES=true. Depending on "degree of corruption" this may work or not. As @ mraedis mentioned, these logs about partially broken images appear regularly. Immich uses "sharp" and "libvips" for image processing: if they emit a warning, we can either skip processing (by default) or force processing (risk of crash, huh, therefore not enabled by default)
- https://docs.immich.app/install/environment-variables/
- https://github.com/immich-app/immich/discussions/23005
Input file is missing: /data/upload/b23def0e-2bdb-499b-b9dc-c4b72e1c21e9/cf/61/cf610399-d3e2-483c-a3c8-168fe2d0aed6.JPGHmm, not sure about these logs. It seems the database knows about assets which are not present on disk anymore. As seen from
[trace] Generate thumbnail for: ... | 200911 IMG_7300.JPG | ... | 2009-11-04T00:00:00+00:00 Immich extracted asset metadata (dateTimeOriginal) successfully. So the file was there on disk at some point in the past. Do you have an idea why they are missing?
Also, you can try a command from https://discord.com/channels/979116623879368755/1435700032644059236/1436674470097064028 to get a list of assets missing on disk.[Discussion] [Feature] Generate thumbnails for broken JPEG images by default (immich-app/immich#23005)
That missing file seems to be one of the svg files that were causing issues
Im not sure if I deleted it or if I swapped to a converted jpg first
That should be it, will look at it tomorrow