Immich Jobs Not running or Failing
Hi everyone, I've been using Immich for some time now without any issues but i've noticed recently that some of my images are failing to load and show as "Error loading image" (see image) so I thought I would manually run the Generate Thumbnails job but it has been active for some time now without completing.
I have tried putting the containers down and spinning them back up again and running the job, sometimes it will get stuck, other times it will complete with no changes.
This has happened to the Storage Template Migration job also.
I have had a look in the Immich_server logs and it is showing Unable to run job handler.
Like I say, I've been using it without fail for a while now and now it seems to be having issues.
I am running the latest v1.131.3 server version.
Any advice would be appreciated
35 Replies
:wave: Hey @347073,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Posting the .env below so you dont need to download it
You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
The location where your uploaded files are stored
UPLOAD_LOCATION=./library
The location where your database files are stored
DB_DATA_LOCATION=./postgres
To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Belfast
The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
Connection secret for postgres. You should change it to a random password
Please use only the characters
A-Za-z0-9, without special characters or spaces
DB_PASSWORD=mysupersecurepassword
The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immichSome additional screenshots of container status - redis logs - and postgres logs
Troubleshooting I have tried so far is:
Have you updated recently? What's the output of
docker ps -a?
Oh sorry, you posted thatI try and stay up to date with the latest releases so I'll update maybe a day or so after its been released
I see you updated to valkey already, when was that?
I updated today, i reviewed the compose yml and made sure everything was using the up to date images from the immich website
Happy to provide any other screenshots or evidence if that would help
Is that the full output of docker ps -a? Want to make sure you don't have any redis still hanging around. Also not sure if any other changes were needed for the valkey update, something seems to be wrong with the job queueing
You could try bring down the stack, removing the redis/valkey volume data, then bring it back up and re-queing a job
In terms of all the containers running, just about fit all the ones running in one screenshot, doesnt seem to be any other Redis ones running
Yeah, I would try bringing down the stack. Deleteing the valkey volume, then bring it back up and try requeuing a job
Before I mess and break something, how would I go about deleting the valkey volume? Taking down and back up the stack I can do pretty easy 😆
Oh you don't have a volume for it. Let me check with the other guys and see
Thank you
you could try re-deploying the stack - not sure if this would help any. I'd suggest ensuring you have backups before proceeding
docker compose down then docker prune then docker compose upHi @Tempest , I have given that a try, run the containers again and tried to run the Storage Template Migration job and receive the same error still
It did delete a lot of old images and saved me a good few GB so thanks for that, but didn't fix this issue unfortunately
If you switch back to redis instead of valkey do things work?
Let me give that a try now
So I have moved the Redis image back to the previous version - see image.
I have then docker compose down - docker system prune - docker compose up
I have done an initial test by changing the Storage Template and running the Storage Template Migration job which still unfortunately comes back with an error
In the UI the job just shows as Active and will remain like that
what version did you upgrade from?
if you click on the version number in the immich web app, can you share the screenshot that pops up?
I can indeed - see below
sorry I don't know how to help you further here.
Was a migration improperly terminated when updating containers? That's about the only thing I can think of. Perhaps restoring a database from the previous backup..
No worries, thank you for your help. I'll mess around with it later on today and see what I can do
@Tempest out of curiosity, If I wanted to completely wipe my server of Immich and start over, how easy would it be to import all my photos that are currently in Immich to a new instance of it?
You could use something like immich-go to upload all assets to a user. The immich CLI would work as well
Okay thanks, will give that a go and see if just starting over again with the new Redis image works
You should be able to restore a backup of the database as well, is that not working for you?
No I tried that but it didnt work and just started from scratch again. I'm not too bothered about the database to be honest, there's only 1 user on there and there's just a load of tags that I have done for images like receipts which I dont mind doing again
@Tempest I have deleted all traces of Immich from my server including the images, volumes, and even changed folder location on my server local drive.
I started up the containers using this docker compose file and env file, I have create a brand new user and effectively set up Immich for the first time.
I have uploaded my first image which is just a random screenshot on the same server Immich is hosted on, and I get this Error loading image and the following error message on immich_server logs.
The image file is also stuck is the Upload folder rather than being transferred to library.
I'm really at a bit of a loss on what to check next...
The error message on the docker immich_server logs referring to Unable to run job handler, is there a specific way I can troubleshoot this? Is is that the immich_server cannot access the redis/or postgres server so it cant process the job, or could it potentially be a permissions thing, although the permissions haven't changed on any of the folders
Just trying to troubleshooting through the problem, it looks like the Immich_Server container is working as it allows upload, it lets me login fine, and let's my upload images, it's then at the point of I believe the jobs where it fails with that error message which I'd assume is the Redis container so could it be an issue that the server cannot see the Redis container so it falls over, or can it see it but Redis doesn't have the permissions to run the microservice job...
What type of file system are you running?
The server is running Ubuntu 24.04 with the latest patches, and the file system is just the standard one that comes with Ubuntu