Immich•3mo ago

Jobs process intermittently, some do and some don't

gene_wood · 2025-10-02T23:46:27.194Z

- All the troubleshooting steps you've tried so far. * I've blown away my immich and started from a new installation and just uploaded a single image. The problem is that it doesn't seem like any microservices are picking up jobs. I've yet to get a single job to process. In the case of a single image upload, the one job of generating thumnails just remains in "Waiting" state. In more complex situations, I've tried uploading many images and all jobs stay in "Waiting". I've also tried enabling an external library and those external library jobs just remain in "Waiting" and nothing processes. - Any recent changes you've made to Immich or your system. * I can reproduce the problem from a brand new immich installation - Details about your system (both software/OS and hardware). * Ubuntu 22.04.5 - Details about your storage * All of the directories, including the UPLOAD_LOCATION of /data/immich/library are on a HDD and the partition is ext4 - The version of the Immich server, mobile app, and other relevant pieces. * immich 2.0.0 - Any other information * The differences from the stock `docker-compose.yml` that I'm using are * I'm following these instructions ( https://docs.immich.app/FAQ/#how-can-i-run-immich-as-a-non-root-user ) which indicate how to run as a non-root user * All of the users configured in the `docker-compose.yml` are owners of the associated directories that they access * I can confirm that redis is writing to it's `dump.rdb` file, postgres is writing to it's files, immich stores the uploaded image in the library directory and the ml container writes files to it's directories. * The immich app itself seems to work fine in most regards, it appears to just be the "Jobs" function. * I also can't figure out how to get any diagnostic data on the microservices part of immich to try to understand why the jobs aren't processing. I see no mention of the jobs or microservice in any logs as if the entire feature is just absent.

All the troubleshooting steps you've tried so far.
- I've blown away my immich and started from a new installation and just uploaded a single image. The problem is that it doesn't seem like any microservices are picking up jobs. I've yet to get a single job to process. In the case of a single image upload, the one job of generating thumnails just remains in "Waiting" state. In more complex situations, I've tried uploading many images and all jobs stay in "Waiting". I've also tried enabling an external library and those external library jobs just remain in "Waiting" and nothing processes.
Any recent changes you've made to Immich or your system.
- I can reproduce the problem from a brand new immich installation
Details about your system (both software/OS and hardware).
- Ubuntu 22.04.5
Details about your storage
- All of the directories, including the UPLOAD_LOCATION of /data/immich/library are on a HDD and the partition is ext4
The version of the Immich server, mobile app, and other relevant pieces.
- immich 2.0.0
Any other information
- The differences from the stock
```
docker-compose.yml
```
  docker-compose.yml
```
docker-compose.yml
```
  docker-compose.yml
  that I'm using are
  - I'm following these instructions ( https://docs.immich.app/FAQ/#how-can-i-run-immich-as-a-non-root-user ) which indicate how to run as a non-root user
  - All of the users configured in the
    docker-compose.yml
    docker-compose.yml
    docker-compose.yml
    docker-compose.yml
    are owners of the associated directories that they access
- I can confirm that redis is writing to it's
```
dump.rdb
```
  dump.rdb
```
dump.rdb
```
  dump.rdb
  file, postgres is writing to it's files, immich stores the uploaded image in the library directory and the ml container writes files to it's directories.
- The immich app itself seems to work fine in most regards, it appears to just be the "Jobs" function.
- I also can't figure out how to get any diagnostic data on the microservices part of immich to try to understand why the jobs aren't processing. I see no mention of the jobs or microservice in any logs as if the entire feature is just absent.

docker-compose.yml4.3KB

env1.32KB

docker-logs-immich_server-nocolor.txt52.16KB

docker-logs-immich_redis-nocolor.txt948B

docker-logs-immich_postgres-nocolor.txt3.96KB

docker-logs-immich_machine_learning-nocolor.txt810B

docker-ps-a.txt2.34KB

Immich•10/2/25, 11:46 PM

:wave: Hey @gene_wood,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.

References

Container Logs:
```
docker compose logs
```
```
docker compose logs
```
docs
Container Status:
```
docker ps -a
```
```
docker ps -a
```
docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

Immich•10/2/25, 11:46 PM

Checklist

I have...

:ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
:ballot_box_with_check: read applicable release notes.
:ballot_box_with_check: reviewed the FAQs for known issues.
:ballot_box_with_check: reviewed Github for known issues.
:ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
:ballot_box_with_check: uploaded the relevant information (see below).
:ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

Your docker-compose.yml and .env files.
Logs from all the containers and their status (see above).
All the troubleshooting steps you've tried so far.
Any recent changes you've made to Immich or your system.
Details about your system (both software/OS and hardware).
Details about your storage (filesystems, type of disks, output of commands like
```
fdisk -l
```
```
fdisk -l
```
and
```
df -h
```
```
df -h
```
).
The version of the Immich server, mobile app, and other relevant pieces.
Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the

/close

/close

command, and re-open it later if needed.

IImmich ## Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest rel...

ImmichAPP•10/2/25, 11:47 PM

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

gene_woodOP•10/2/25, 11:56 PM

Here's the full contents of the immich directories showing the permissions and the various immich users (ml, postges, etc) writing their files as themselves.

And I can confirm that the uploaded image is present because, though it has no thumnbail (Just says "error loading image") and it has no rendered image (same error), if I click the download button for the image, I can download it just fine.

immich-ls-alR.txt102.65KB

gene_woodOP•10/3/25, 12:38 AM

Is the immich process which executes jobs a distinct process from immich-apiimmich-api? Or does immich-apiimmich-api work with the redis container to process jobs? I ask because in the immich container, I only see an immich-apiimmich-api process running, but nothing mentioning microservices or a worker process that would specifically do job processing.

Sergey Katsubo•10/3/25, 6:08 AM

Thank you for posting detailed info, this's really helpful.

Re: job logging. You already enabled VERBOSE, great. Unfortunately jobs typically do not log anything unless there are errors. There is room for improvement it seems.
You can check JSON with status of queues at the jobs GET endpoint: https://my.immich.app/api/jobs (assuming you are already logged in)

Re: immich processes. This is how it looks like from my Debian host:

immichimmich = worker
`immich-api` = api```pstree -pc ├─containerd-shim(3486341)─┬─tini(3486360)───immich(3486412)─┬─immich-api(3486490)─┬─{immich-api}(3486492) ... │ │ │ ├─{immich-api}(3486508) │ │ │ └─{immich-api}(3486509) │ │ ├─{immich}(3486419) ... │ │ ├─{immich}(3486456) │ │ └─{immich}(3486491)```

Sergey Katsubo•10/3/25, 6:11 AM

Re: internal docker networks.
Best practice is to expose only Immich to web/proxy network while keeping other components on the default network. To avoid possible interference with other containers in the network.

services:
  immich:
    networks:
      - web_custom
      - default

services:
  immich:
    networks:
      - web_custom
      - default

(Same for other services to make web_custom clean and containing only HTTP services)

Sergey Katsubo•10/3/25, 6:15 AM

The behavior you are observing is typical in case of several Redis containers present in the Immich network and having the same "redis" name. So they are clashing, Immich (docker / internal dns) distributes requests among them, leading to unpredictable job processing.
You can check IP addresses in network dump and see if there appear different destination IPs for Immich->Redis requests

docker run --rm --net container:immich_server nicolaka/netshoot tcpdump -n 'port 6379'

docker run --rm --net container:immich_server nicolaka/netshoot tcpdump -n 'port 6379'

Obviously you can check

docker ps -a

docker ps -a

for other Redis instances.

Sergey Katsubo•10/3/25, 6:19 AM

Similar:

help-desk-supportThumbnails and Previews not working "a lot of jobs queuing up but never starting. I have Authentik running on the same network"
https://github.com/immich-app/immich/discussions/20548#discussioncomment-14082097 "it is the proxy manager network and makes exposing public resources easier. So there are many things on it, including other instances of postgres and redis containers"

Sergey Katsubo•10/3/25, 6:34 AM

Also it's possible to check keys in Redis with either redis-cli or redis UI. (There are also specific "bullboard" UIs for viewing BullMQ status as Immich uses bullmq under the hood; I use venatum/bull-boardvenatum/bull-board).

See below "redis-cli" command adapted from the abovementioned github thread.

Trigger the thumbnail geeneration job from Jobs page.

Then run the command against immich_redisimmich_redis and other redis containers (if any are present).

REDIS=immich_redis
docker exec "$REDIS" redis-cli --raw XREVRANGE "immich_bull:thumbnailGeneration:events" + - COUNT 30 | \
awk '/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "ID=" $0 ; next }
{ if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } }
END { print output }'

REDIS=immich_redis
docker exec "$REDIS" redis-cli --raw XREVRANGE "immich_bull:thumbnailGeneration:events" + - COUNT 30 | \
awk '/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "ID=" $0 ; next }
{ if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } }
END { print output }'

Example output:

ID=1759449605350-1, event=drained
ID=1759449605350-0, event=completed, jobId=239, returnvalue=null, prev=active
ID=1759449600735-0, event=completed, jobId=238, returnvalue=null, prev=active
ID=1759449600220-0, event=active, jobId=239, prev=waiting
ID=1759449600216-1, event=waiting, jobId=239
ID=1759449600216-0, event=added, jobId=239, name=AssetGenerateThumbnails
ID=1759449600117-0, event=active, jobId=238, prev=waiting
ID=1759449600107-1, event=waiting, jobId=238
ID=1759449600107-0, event=added, jobId=238, name=AssetGenerateThumbnailsQueueAll

ID=1759449605350-1, event=drained
ID=1759449605350-0, event=completed, jobId=239, returnvalue=null, prev=active
ID=1759449600735-0, event=completed, jobId=238, returnvalue=null, prev=active
ID=1759449600220-0, event=active, jobId=239, prev=waiting
ID=1759449600216-1, event=waiting, jobId=239
ID=1759449600216-0, event=added, jobId=239, name=AssetGenerateThumbnails
ID=1759449600117-0, event=active, jobId=238, prev=waiting
ID=1759449600107-1, event=waiting, jobId=238
ID=1759449600107-0, event=added, jobId=238, name=AssetGenerateThumbnailsQueueAll

gene_woodOP•10/3/25, 9:12 PM

@Sergey Katsubo !!! Fantastic. As I'd done a bunch of troubleshooting and couldn't come up with the solution, I didn't have high hopes for asking here, but you got it on the first try. You're absolutely right that I'm running a second redis instance (for Authelia) in the same network and Immich was interacting with two redis instances intermittently. Before I improve my docker network setup, my fix was to add REDIS_HOSTNAME=immich_redisREDIS_HOSTNAME=immich_redis to my Immich

.env

.env

file so that Immich uses the defined container name of immich_redisimmich_redis instead of just redisredis which has the name collision. (I also confirmed that Authelia was already using a dedicated name of authelia_redisauthelia_redis instead of redisredis so it wasn't experiencing the same problem). Thank you soo much for figuring this out for me! Do you think it's worth me PRing a new FAQ question answer about this intermittent job pickup symptom in case someone else encounters this?

Sergey Katsubo•10/3/25, 9:40 PM

Do you think it's worth me PRing a new FAQ question answer about this intermittent job pickup symptom in case someone else encounters this?

Sounds great!

Jobs process intermittently, some do and some don't

References

Checklist

Information

Similar Threads

Similar Threads

References

Checklist

Information

Similar Threads