Immich reporting "Unable to set queue concurrency, worker not found..."
I do not know when exactly this issue started, but I had been on 1.123.0 for the last few weeks and everything was seemingly working fine. I have a Load Balancer setup through NGINX and the machine learning settings have been updated to point directly to it. I have 2 other VM's on other machines that are running just the machine learning containers and it was working as expected. If I try to set or change the Job concurrency settings, it gives me the errors mentioned. Additionally, if I now try to kick off a job, I get significantly more errors, seemingly for each of the files that it tries to perform work on. (errors attached)
I checked the logs today for Immich and started seeing the below errors. I upgraded Immich to 1.124.2 but am continuing to see the following errors regarding the jobs.
Here is my NGINX load balancer config:
resolver 10.0.1.52 10.0.1.53;
upstream immich_ml_containers {
server prd-docker08:3003 max_fails=3 fail_timeout=15s;
# server prd-docker09:3003 max_fails=3 fail_timeout=15s; ## main immich server # Disabled to ensure main server does not go down
server prd-docker11:3003 max_fails=3 fail_timeout=15s;
# server prd-docker13:3003 max_fails=3 fail_timeout=15s;
# server lovo:3003 max_fails=3 fail_timeout=15s;
# server dominator:3003 max_fails=3 fail_timeout=15s;
}
server {
listen 80;
location / {
proxy_pass http://immich_ml_containers;
}
}
docker-compose.yaml attahced
7 Replies
:wave: Hey @ThorsHammer,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJAChecklist
I have...
1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.GitHub
immich-app immich · Discussions
Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.
FAQ | Immich
User
GitHub
Issues · immich-app/immich
High performance self-hosted photo and video management solution. - Issues · immich-app/immich
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Can you post the output of
docker ps -a?jobs logs example
As an update, I have also started a machine learning container, updated the machine learning URL to http://immich_machine_learning:3003 since it's running on the same docker network as immich_server... yet I'm still seeing the same WARN messages about being unable to set concurrency, worker not found.
This tells me that this is not a load balancer issue on my end.
So I played around quite a bit ...
Turns out that I believe the Warnings about setting concurrency and worker not found are something i can ignore.
Additionally, with the connection timeouts that I was seeing, I believe that this is due to having too high of a concurrency rate and not enough processing power.
I dropped the concurrency rate to 4 (have 4 PC's working through) it seems to give way less connection timeout errors. They still happen, but less frequently.