I
Immich•5mo ago
Devilkin

Machine learning not working

Hi, I've had an Immich instance running for a while now, and everything was working well. Facial recognition worked, i could search, etc. Everything in CPU, no hardware acceleration available, but it worked. Yesterday I noticed that search didn't work anymore, and new faces weren't being added. Some searching led me to the following error in the immich_server docker container: 
[Nest] 17  - 12/29/2024, 10:51:29 AM    WARN [Api:MachineLearningRepository~gl5z9lta] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
[Nest] 17  - 12/29/2024, 10:51:29 AM   ERROR [Api:ErrorInterceptor~gl5z9lta] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:41:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:64:26)
    at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:66:27)
[Nest] 17  - 12/29/2024, 10:51:29 AM    WARN [Api:MachineLearningRepository~gl5z9lta] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
[Nest] 17  - 12/29/2024, 10:51:29 AM   ERROR [Api:ErrorInterceptor~gl5z9lta] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:41:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:64:26)
    at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:66:27)
which occurs everytime I try to search for anything. I've tried re-spinning everything up (deleting the docker container images even), I've removed the model cache volume, but the problem persists.

docker-compose.yml

env

immich_machine_learn...

immich_server.log

8 Replies
Immich
Immich•5mo ago
:wave: Hey @Devilkin, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy). 6. :ballot_box_with_check: uploaded the relevant information (see below). 7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.
Devilkin
DevilkinOP•5mo ago
$ docker ps -a
CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS                   PORTS                                          NAMES
8d9419a63cd5   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash s…"   2 minutes ago   Up 2 minutes (healthy)   127.0.1.1:2283->2283/tcp, ::1:2283->2283/tcp   immich_server
1ff097ed7f31   tensorchord/pgvecto-rs:pg14-v0.2.0                   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)   5432/tcp                                       immich_postgres
da8eef8c008f   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     2 minutes ago   Up 2 minutes (healthy)                                                  immich_machine_learning
31c1a1afa713   redis:6.2-alpine                                     "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp                                       immich_redis
$ docker ps -a
CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS                   PORTS                                          NAMES
8d9419a63cd5   ghcr.io/immich-app/immich-server:release             "tini -- /bin/bash s…"   2 minutes ago   Up 2 minutes (healthy)   127.0.1.1:2283->2283/tcp, ::1:2283->2283/tcp   immich_server
1ff097ed7f31   tensorchord/pgvecto-rs:pg14-v0.2.0                   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)   5432/tcp                                       immich_postgres
da8eef8c008f   ghcr.io/immich-app/immich-machine-learning:release   "tini -- ./start.sh"     2 minutes ago   Up 2 minutes (healthy)                                                  immich_machine_learning
31c1a1afa713   redis:6.2-alpine                                     "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp                                       immich_redis
The machine has an Intel(R) Xeon(R) CPU E3-1230 v5 @ 3.40GHz - old, but it did work. Currently assigned 6gb ram for immich, which it's hardly using anything now.
Immich
Immich•5mo ago
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
Devilkin
DevilkinOP•5mo ago
I've tried issuing a curl from the immich_server container towards the immich-machine-learning container on port 3003, but that doesn't seem to work either: 
$ docker exec -ti immich_server /bin/bash 
root@8d9419a63cd5:/usr/src/app# 
root@8d9419a63cd5:/usr/src/app# curl http://immich-machine-learning:3003/ -vv
*   Trying 172.19.0.3:3003...
* connect to 172.19.0.3 port 3003 failed: Connection refused
* Failed to connect to immich-machine-learning port 3003 after 0 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to immich-machine-learning port 3003 after 0 ms: Couldn't connect to server
$ docker exec -ti immich_server /bin/bash 
root@8d9419a63cd5:/usr/src/app# 
root@8d9419a63cd5:/usr/src/app# curl http://immich-machine-learning:3003/ -vv
*   Trying 172.19.0.3:3003...
* connect to 172.19.0.3 port 3003 failed: Connection refused
* Failed to connect to immich-machine-learning port 3003 after 0 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to immich-machine-learning port 3003 after 0 ms: Couldn't connect to server
So the container is there, it's resolveable, but connectivity seems to be null.
bo0tzz
bo0tzz•5mo ago
Can you post the logs from the ml container? Nvm you did they just don't show nicely on my phone lol
Devilkin
DevilkinOP•5mo ago
I've already included them. There's nothing beyond that point (and I didn't find how to put that in verbose mode)
bo0tzz
bo0tzz•5mo ago
I'll take a look when I'm at my pc
Devilkin
DevilkinOP•5mo ago
Hah. Found it. It seems that IMMICH_PORT is being used for the immich-machine-learning container also? ergo it was listening on 2283 seems a bit counter-intuitive that (Also, I don't remember why I ever specified that port in the .env. I had a reason, but... whatever it was, I don't recall)

Did you find this page helpful?