(Weird Bug) Job Handler stuck and instead increases the counter for every click to run a job.
The problems began following a recent update (around v1.42.0).
Background: The first issue arose after uploading new 4K HDR (HEVC 10-bit BT.2020) videos from a Pixel 8 Pro, which are significantly more VRAM-intensive to decode than my previous SDR content.
When the system ran the default 8 concurrent transcodes (the driver limit), the GPU rapidly ran out of memory, generating continuous CUDA_ERROR_OUT_OF_MEMORY errors. Then the transcoded videos that failed had corrupted metadata, resulting in "borked" aspect ratios (near 1:1) and incorrect playback speeds (200% speed on mobile).
I have since lowered the concurrent transcoding limit to 4 for HDRs. The transcoding pipeline is now successfully processing the 4K HDR content, and this specific VRAM-related issue is resolved.
Current Critical Issue - Stuck/Non-Executing Job Handlers
During the troubleshooting of the VRAM issue, I discovered a persistent problem with the background job handlers.
The Metadata Extraction job handler appears stuck (hence the 1:1 videos on mobile).
Clicking the "Missing" button for the Metadata Extraction job does not initiate the job. Instead, it simply increments the "Job Waiting" counter for every single click, even though the job never begins execution. (Check attached GIF)
Then I checked other job handlers (e.g., External Libraries, Thumbnails, etc.) and found that almost all exhibit the same behavior: they increment the "Job Waiting" counter without starting the job.
I performed a full system reset using a clean docker prune followed by a fresh container startup. The job handler behavior persists. Some handlers might execute once, but the majority default back to the "increment counter without running" state.
I need help in diagnosing this issue, which appears to be independent of the previous VRAM problem and is preventing Immich from extracting metadata and completing other background tasks.
8 Replies
:wave: Hey @Seven7Lemons,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.Docker Compose and Env files
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Immich Server Logs
Remove the networks from your yaml file and retry. Generally this is because of networking issues
I removed the network and tried again : same results, I can reach all the microservices from immich server shell,
root@98f12899a29b:/usr/src/app# curl http://immich_postgres:5432 curl: (52) Empty reply from server root@98f12899a29b:/usr/src/app# curl http://immich_redis:6379 curl: (52) Empty reply from server root@98f12899a29b:/usr/src/app# exitAgain no errors from the logs, also troubleshooted my firewall: That is not the problem either
I've seen
Template not initialized recently and what actually was happening was one of the containers was constantly restarting
You've only posted immich server logs, but can you do the full stack?Microservices logs, hoping you meant the other container logs