Thumbnails not generating
Hi, I cannot see thumbnail images that my partner uploaded from her iPhone, they always look as in the attached pic.
Setup
We both see each other's pictures on our timeline as partner accounts. To date, I only imported our old pics from Google Photos and the thumbnails are all there.
- My wife has an iPhone with the immich app (v1.135.1 build.210) installed.
- I have an Android phone with the immich app (v1.135.1 build.202) installed.
- the Web app is up to date
- Immich v1.135.3 is hosted on my Ubuntu server with latest updates, it sits behind a Caddy reverse-proxy.
- Attached are my docker-compose.yml and .env file.
the issue
If I take a picture, this behavior is observed:
- on my android app:
- the thumbnail is shown
- the full image is shown
- on the web with my account:
- the thumbnail is broken
- the full image is shown
- on my wife's iPhone app:
- my wife cannot see the thumbnail
- my wife cannot see the full image
- on the web with my wife's account:
- my wife cannot see the thumbnail
- my wife can see the full image
If my wife takes a picture -> same issue as above but reversed
What I tried
- "docker ps" -> all healthy
- "docker compose down" -> "docker pull" -> "docker compose up -d"
- manually started "Generate thumbnails" job
- manually started all other possible jobs ("Smart search" took some time but finished)
- adjust the docker-compose to not assign the network "web_proxy_network" to the redis and database container (i.e. comment out the lines "networks:" and " - web_proxy_network" for both)
key question
How can I see my partner's pictures as thumbnails and full images, both on the Web and on my Android App?
PS: If you need logs, pls specify exactly how I can get the relevant ones (i.e. do I need to first "compose down" & "compose up" before calling them?
thx a MILLION in advance for any pointers and tips
107 Replies
:wave: Hey @BenjaminSchumann,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.What is this?
These aren't real immich env vars
thx, I use these elsewhere to define the custom URL to access Immich
Did you also set the URL in the admin section?
Do you mean here? No. Should I?
Shouldn't be necessary, just checking
what does the immich thumbnail say it's location is?
Ah right maybe difficult if you look at it in the app 🤔
Does it exhibit the same behaviour in the web version?
On the web, I cannot see the thumbnails from my partner and also not from myself. Pic here shows 1 pic from my partner's phone and the other from my phone:
ah also not from yourself
Are the jobs still running?
man I really need to read your topic better
Anyway
If you rerun the job for a specific image, you should see it appear in the logs
Are the jobs still running?no, not as far as I can see. No idea why there are remaining "Migration" jobs...
If you rerun the job for a specific image, you should see it appear in the logsHow can I do this?
on the asset page -> three dot menu -> refresh thumbnails
it should either pop up or show an error why
ok, thx. How can I get the log for that? Simply "sudo docker logs immich_server"?
Either that or
sudo docker compose logs immich-serverIf I do "sudo docker logs --since 5m immich_server" after clicking "Refresh thumbnails" on the broken thumbnail, I get:
[Nest] 17 - 07/02/2025, 2:26:31 PM LOG [Api:EventRepository] Websocket Connect: fBMOtpEFEo1EDlTzAABL
[Nest] 17 - 07/02/2025, 2:26:52 PM LOG [Api:EventRepository] Websocket Disconnect: fBMOtpEFEo1EDlTzAABL
[Nest] 17 - 07/02/2025, 2:27:07 PM LOG [Api:EventRepository] Websocket Disconnect: OKCNugGUoF5xdoCeAABJ
[Nest] 17 - 07/02/2025, 2:27:10 PM LOG [Api:EventRepository] Websocket Connect: u6QIhr1yyJU5jYiUAABN
[Nest] 17 - 07/02/2025, 2:28:59 PM LOG [Api:EventRepository] Websocket Disconnect: u6QIhr1yyJU5jYiUAABN
[Nest] 17 - 07/02/2025, 2:29:03 PM LOG [Api:EventRepository] Websocket Connect: uHNDz3mcEh6-QwIrAABP
So absolutely nothing 👀
I did click some more "refresh thumbnail" and "refresh metadata" on the web... Now it does show the thumbnail for my picture. Still nothing for my wife's:
so at least 1 step further, thx 🙂
not sure you can refresh on shared assets
nope
Awesome, refreshing them from my wife's accoutn worked. Let me test if it auto-creates them for new pics...
But from the outcome here it seems to me like some job(s) were stuck
Agreed. Check the jobs screenshot above: I still have 4 "Migration" jobs waiting, no idea why
Also, taking new pictures still creates the same issue: thumbnails are not created and I cannot see my wife's pics and vv
You could compose down, delete the redis/valkey volume, docker compose up and it will have killed all cached jobs
Thx, I read this in a different post, happy to try. However, how exactly do I "delete the redis/valkey" volume?
That'd be up to you, you can get the volume attached to a container from the CLI
For instance
Then you can just check:
and use
docker volume rm xyzthx a ton, really useful. I did this:
1. docker compose down --volumes
2. checked that redis vol is gone (yes)
3. docker compose up -d
Now the jobs page has changed (see pic) but it is not actually progressing at all. Also, still not creating thumbnails automatically for new pics (even those taken after the "compose up" call) :/
Any other ideas?
what does
docker compose logs tell you
It is downloading the ML model?
Because you've removed that with --volumestons of entries like this. What does it mean?
(note that these are older pics that show and work fine)
That just means it's trying to upload something that already exists
ah, probably because I removed all data from my app and now it tries to reupload my phone files.
any idea why my jobs are now stuck as above?
.
Is that because I removed the ML model? How can I check this? "docker ps" shows all "healthy", including the ML
the ml container logs should show some message about downloading the model
once done, should the redis jobs ... start?
No that's when the facial jobs should be finishing up
are they not?
thx again. I now have 1000s of different jobs waiting (didn't touch anything these past few hrs). Why would that be, how can I trigger them and is there something wrong with my redis setup in the docker-compose (see original post)?
see ML logs from the last 4 hrs below:
" ERROR Worker (pid:122) was sent SIGINT!" ...?
That's normal, should really be an INFO rather than error
Logs seem good, loaded the face algo and then the smartsearch algo
still got the same issue. Thumbnails not shown...
I just did "docker restart immich_server" and now it shows just 1 GENERATE THUMBNAIL job waiting (and 1 waiting for STORAGE TEMPLATE MIGRATION for whatever reason)...
But still no thumbnails generated 😦
Any other ideas?
I switched to VERBOSE logging and did sudo docker logs immich_server --since 5m | grep -E "ERROR|WARN|VipsJpeg|thumbnail|AssetJobService|AssetService|JobService"
Anything useful you can see below? (Gemini LLM interprets it as normal...)
All I see is it generating a lot of thumbnails :p
thx. btw, just purchased a supporter, it is really great to have this kind of help, thx a ton.
So what you are saying is: "it all looks good", right? Why would I not see those thumbnails then?
Is it possible (for money) that you or someone else from the team helps me debug this live via a screenshare at some point? I really want to make immich work 🙂
Also, if the logs show that it is generating a lot of thumbnails: why is the jobs page stale? (the "waiting" for "GENERATE THUMBNAILS" went to 3 because I clicked on the "all" and "missing" button next to it... it just increases the counter but nothing ever starts
Very curious indeed
You can tell from the logs it's always a new round of assets that get their thumbs
What is immich running on, what type of storage etc?
installed onto a self-hosted Ubuntu server. The data lives on a different SSD (M2) than the basic docker container and the Ubuntu distribution itself.
This worked fine so far, until my wife started sharing her pics from her phone 😦
so "fdisk -l" shows:
Disk /dev/nvme0n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Samsung SSD 990 EVO Plus 2TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: 512GB SSD
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: FEA93AEF-1E92-45C5-BD2B-4D68A167F8AF
Immich container are on the latter, the file DB is on the former (see original docker-compose and .env file)
"df -h" results attached
Immich docker container sit on the /dev/sda2/ and the file DB is on the "/dev/nvme0n1"
updated Ubuntu 2 days ago...
Also, I followed your advice on "docker volume rm" for the redis volume (so compose-down, rm the redis and then compose-up)...
Now there are no jobs waiting or active at all. However, the server itself seems to work really hard (I can hear it)... thumbnail state as before....
you could check with (b/h)top what the process is that's making it churn
How do you know that no thumbnails are generated / what method are you using to verify this?
no jobs running -> gui thumbs still broken after running "missing"
So the thumbs are not generated or not served to the client?
Hey, thx for getting back to me, both 🙂
you could check with (b/h)top what the process is that's making it churnThe server ran heavily all night but is now finished, so
btop does not help anymore
How do you know that no thumbnails are generated / what method are you using to verify this?Not sure if they are generated or not but I can see the broken "Error loading image" thumb on the Web. Interestingly, it is now showing correctly on my app so some thumbnail stuff seems to have happened...
no jobs running -> gui thumbs still broken after running "missing"Huh? Not sure what you mean? After a full night of heavy server activity (it is now done), the jobs dashboard shows only 1 outstanding "GENERATE THUMBNAIL" job. As usual, if I click either "All" or "Missing" next to it, the "Waiting" counter simply increases but no job is ever started. I attach the logs when I click them.
So in a nutshell:
- no thumbnail shown from my wife's phone in my Web account (but I can see it now on my phone app)
- logs as above
- still the weird "jobs" behavior: the "Waiting" count just grows when clicking "All" or "missing"
Is this pointing to some deeper redis issue, by any chance? Why is it not starting those jobs, why does it keep getting stuck (seemingly)?
(When I take a new picture with my phone, I can see a new "Extract metadata" job but it is also stuck in "Waiting" and I cannot see a thumbnail on the web...)
(right-clicking on those new images and forcing "Refresh thumbnail" also just adds to the "Waiting"...)
If you pull down twice on your wife's phone, does it still show missing?
If you pull down twice on your wife's phone, does it still show missing?Yes, still missing, same on my phone (for my own pictures taken previously...)
Could this be caused by the Tailscale setup I employ? I recently set this up such that devices in the VPN network use the server to resolve DNS queries for the URL of immich (and other services I host). Setup as attached.
Would be weird since immich does work fine (upload of files...) but maybe redis has issues here?
Is my docker-compose.yml correctly set for redis, especially it using the same network as the other services?
I would expect resolve/unreachable errors if inter-container communication was the issue
More probable is that the app has trouble following different IPs
More probable is that the app has trouble following different IPsNot sure what you mean, how can I test for that? Also note that I cannot see the thumbnails in the web as well. Their "GENERATE THUMBNAIL" jobs are all stuck (7 jobs "waiting"). How can I force redis to start those jobs (ideally without force-deleting the volume again, as that triggered 16 hrs of heavy server work, which I do not want to trigger each time...)
It shouldn't be triggering any additional work at all, at most what it should do is try to process the missing items
I suggest you first test the stock compose without the tailscale network changes, and try direct IP at home for the app, this should rule out that at least
I suggest you first test the stock compose without the tailscale network changes, and try direct IP at home for the app, this should rule out that at leastSorry, what do you want to rule out with this? What do you mean by "try direct IP at home for the app"?
To rule out any tailscale trouble you should connect to the "normal" IP at home, so no VPNs no proxies nothing in between
so... you mean to use the default immich "docker-compose.yml" and ".env", right?
But how would my app connect to that at all? And would I not need to adjust all those DB locations to match my current variables? (apologies, not feeling 100% confident with Linux yet, thx for any pointers)
the idea is to use a complete stock setup, so that you know if it's a change you made or not
Thx. It did work 100% fine until last week so it is very likely to be caused by those Tailscale changes, tbh.
Say the local setup would work (I have no reason to believe otherwise), what would be another idea to check? Why does the redis server jobs queue not do anything?
Attached the current immich_server VERBOSE logs...
redis logs have not done anything these past few hrs albeit the jobs page showing outstanding jobs 😦
redis logs look like this and, interestingly, also looked like this last night where the server wrokked really hard on something (after deleting the redis volume and restarting the container...)
Also maybe helpful: I just deleted the cache and data for my immich android app, then logged in again fresh. It did (!) show the thumbnails, both for my and my wife's pics!!!
So the thumbnails are there (somewhere). But somehow, they get lost again after just opening an image... Still no thumbnails on the web at all (even in inkognito mode, no cached stuff...)
are there any network errors in your browser? (hit F12 while browsing in chrome)
yes, in fact there are, see attached. Do they mean something to you?
(btw, mobile phone seems to work now but thumbail job queue is still stagnant)
its not found. Looks like a networking issue. Does it work if you connect to http://local_ip_address:2283 ?
"asset media not found"... if I click on the "Response" tab, see above
at the time of this log, is there an error logged simultaneously in immich_server?
its not found. Looks like a networking issue. Does it work if you connect to http://local_ip_address:2283/ ?No, I use Caddy so not opening any ports except 80 and 443. Connection times out
at the time of this log, is there an error logged simultaneously in immich_server?I think so, see attached
on a local network opening the port shouldnt be an issue. To be clear I do not suggest doing this remotely
not sure I follow. My immich is hosted on an Ubuntu machine with no UI. I SSH into it from my Windows desktop. I tried to http://...:2283 on my local desktop and that times out.
Attached is my Caddyfile, if that helps
can you change
to
ok, so first "docker compose down", then change by adding the "ports" section and then "compose up -d" again, then recheck on my Windows browser if I can access locally? on it
ok, done. I can now access it locally as you suggested. Still exactly the same browser error
so... not a network/internet/tailscale/caddy/proxy issue then, I assume?
is your local network 100.121.165.x?
it is the IP of my server...
locally?
as in your computer has the same starting ip address?
it looks like a cgnat ip?
should show you ip addresses used by the server (if you enter it by terminal)
ok, thx. I found 192.168.2.74 using "hostname -I". Using that, I get the same error:
if you click on one of those thumbnails, and choose re-generate thumbnail, what does the server log say?
does the thumbnail appear on page refresh, or is it the same issue?
It does appear, yes. This trick also worked yesterday (thx again).
Weirdly, this also seems to have removed the jobs queue... while it did not trigger that when doing it from the web-version... any idea why?
Now I just took a new picture from my phone and synced it and it works
AWESOME, thx a ton. But what is the lesson here, what was the root cause?
Should I keep the "ports :2283" in my docker-compose?
And how can I fix this in the future? Go back to the local version and manually trigger a thumbnail?
Should I keep the "ports :2283" in my docker-compose?
not required, I don't think. I have it exposed so I'm not sure if the networks causes unintended consequences or not
I'd guess there's a corrupted asset or permissions or something causing your queue to hang, but I could be wrong. Doing stuff differently just helps point to the root cause. Pressing "missing" on the thumbnail job should make all assets visible again as a thumbnail. if that fails perhaps it's hanging on an asset?
well, the prior queue (which just grew with every "missing" click) is now gone since I triggered it from the local site. But I did it from the web-site previously and that just added to the jobs...
Really weird. Would be great to have a UI feature on the jobs page to "flush all waiting jobs" and clean the redis queue entirely...
I think docker down and up clears the queue
perhaps there is some issue with the way the networks are setup with your compose file keeping redis from properly communicating?
yes, this is my hypothesis as well. I removed the "ports 2283" stuff, restarted immich and now it is back to "no thumbnails on the web" (unless I manually click to update them).
Also, no jobs queued at all...
This is so weird, any ideas how I could review my network setup?
docker network inspect
Not necessarily fixing it but generally you shouldn't just throw all your containers into your proxy network. You should only put the container in there that actually needs to be accessible by the proxy (i.e., server). For internal communication you'd create a new network that's juts for internal communication
For Immich for instance I have
and then server has
That being said, this is just good practice but shouldn't change the way things work in general
Unless you have this many containers all in that single network that you're running out of ip addresses (out of the DHCP range). You'd see that with the network inspect thoughthx. Do you user "internal: driver: bridge" on all immich containers? And does it need to be defined anywhere else except under the "networks" section?
Yes I use that network for all Immich containers, server then additionally also has the proxy network
And no, the network section defines it already. This is just the minimal config which is perfectly fine for this use case though
Shouldn't you know this? 😅 You're using a custom network as well
This is basic docker compose networking though so it's probably helpful to read the docs :) https://docs.docker.com/compose/how-tos/networking/
hey, did some more digging. Whenever a new photo is uploaded, it is stored into "/home/data_disk/immich_data/library/upload/some-folder/new-image.jpg" (since my .env file has UPLOAD_LOCATION=/home/data_disk/immich_data/library)
However, when Immich records the location of this new file in its database, it incorrectly adds an extra /upload/ segment. So, the database thinks the file is at upload/upload/some-folder/new-image.jpg.
Then, when Immich tries to create a thumbnail, it checks the path stored in its database (upload/upload/some-folder/new-image.jpg). Because the database path has an extra /upload/ that isn't there on your actual disk, Immich can't find the original image file. This leads to the "Asset media not found" error, so it can't process the image or create a thumbnail.
Is there anything truly wrong with my docker-compose or .env file or why would this happen at all? Uploaded latest files for convenience
Nothing incorrect about upload/upload
I admit it is confusing
Believe it or not but changing the docker-compose to only let immich_server have access to the external network seems to have fixed it (all other containers share a new internal docker network with immich_server now). Fingers crossed this holds.
HUGE thank you to all involved. As I said: I already purchased a license and dont regret it one bit 🙂
@BenjaminSchumann may I know how you solve the issue? I'm running into the exactly same issue after I upgraded from v1.131.3 to v1.139.4 last night.
All newly uploaded photos have no thumbnail, but videos are fine
Same happens to me. It also throws an error on some javascript module not found
I am also facing the same issue
I downgraded to 138.1 and thumbnails are generated correctly again. Must be a bug in the 139 release.
There is a bug for generating thumbnail on RPi on 1.139
@Carl how do i downgrade ?
Reply was given to me in another thread: https://discord.com/channels/979116623879368755/1409934725367201933/1409942694075170957
@Carl thankyou