Immich•2y ago

Facial Recognition ECONNREFUSED

Running a full "Recognize Faces" job floods the docker logs with errors. What I've tried: - stopping the stack & starting it again - using the latest docker-compose straight from GitHub (only adjusted the external mapping of immich-proxy) The errors look like this:

immich_microservices     |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
immich_microservices     | [Nest] 1  - 05/20/2023, 7:35:48 PM   ERROR [FacialRecognitionService] Unable run facial recognition pipeline: 6a173c8f-0a91-4e90-a502-1a73aa01bb1e
immich_microservices     | Error: connect ECONNREFUSED 172.18.0.5:3003
immich_microservices     |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
immich_microservices     | [Nest] 1  - 05/20/2023, 7:35:48 PM   ERROR [FacialRecognitionService] Unable run facial recognition pipeline: 8a6a8e36-a869-4eb5-bc0e-213e517c8633
immich_microservices     | Error: connect ECONNREFUSED 172.18.0.5:3003
immich_microservices     |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)

immich_microservices     |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
immich_microservices     | [Nest] 1  - 05/20/2023, 7:35:48 PM   ERROR [FacialRecognitionService] Unable run facial recognition pipeline: 6a173c8f-0a91-4e90-a502-1a73aa01bb1e
immich_microservices     | Error: connect ECONNREFUSED 172.18.0.5:3003
immich_microservices     |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
immich_microservices     | [Nest] 1  - 05/20/2023, 7:35:48 PM   ERROR [FacialRecognitionService] Unable run facial recognition pipeline: 8a6a8e36-a869-4eb5-bc0e-213e517c8633
immich_microservices     | Error: connect ECONNREFUSED 172.18.0.5:3003
immich_microservices     |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)

There's one for every image it seems to try and scan. The docker compose ps looks like this:

root@immich-ct:~# docker compose ps     
NAME                      IMAGE                                                COMMAND                  SERVICE                   CREATED              STATUS              PORTS
immich_machine_learning   ghcr.io/immich-app/immich-machine-learning:release   "python src/main.py"     immich-machine-learning   About a minute ago   Up About a minute   
immich_microservices      ghcr.io/immich-app/immich-server:release             "/bin/sh ./start-mic…"   immich-microservices      About a minute ago   Up About a minute   3001/tcp
immich_postgres           postgres:14                                          "docker-entrypoint.s…"   database                  About a minute ago   Up About a minute   5432/tcp
immich_proxy              ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   immich-proxy              About a minute ago   Up About a minute   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp
immich_redis              redis:6.2                                            "docker-entrypoint.s…"   redis                     About a minute ago   Up About a minute   6379/tcp
immich_server             ghcr.io/immich-app/immich-server:release             "/bin/sh ./start-ser…"   immich-server             About a minute ago   Up About a minute   3001/tcp
immich_typesense          typesense/typesense:0.24.0                           "/opt/typesense-serv…"   typesense                 About a minute ago   Up About a minute   8108/tcp
immich_web                ghcr.io/immich-app/immich-web:release                "/bin/sh ./entrypoin…"   immich-web                About a minute ago   Up About a minute   3000/tcp

root@immich-ct:~# docker compose ps     
NAME                      IMAGE                                                COMMAND                  SERVICE                   CREATED              STATUS              PORTS
immich_machine_learning   ghcr.io/immich-app/immich-machine-learning:release   "python src/main.py"     immich-machine-learning   About a minute ago   Up About a minute   
immich_microservices      ghcr.io/immich-app/immich-server:release             "/bin/sh ./start-mic…"   immich-microservices      About a minute ago   Up About a minute   3001/tcp
immich_postgres           postgres:14                                          "docker-entrypoint.s…"   database                  About a minute ago   Up About a minute   5432/tcp
immich_proxy              ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   immich-proxy              About a minute ago   Up About a minute   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp
immich_redis              redis:6.2                                            "docker-entrypoint.s…"   redis                     About a minute ago   Up About a minute   6379/tcp
immich_server             ghcr.io/immich-app/immich-server:release             "/bin/sh ./start-ser…"   immich-server             About a minute ago   Up About a minute   3001/tcp
immich_typesense          typesense/typesense:0.24.0                           "/opt/typesense-serv…"   typesense                 About a minute ago   Up About a minute   8108/tcp
immich_web                ghcr.io/immich-app/immich-web:release                "/bin/sh ./entrypoin…"   immich-web                About a minute ago   Up About a minute   3000/tcp

48 Replies

Latte macchiatoOP•2y ago

Oh, and the logs of immich-machine learning look like this:

root@immich-ct:~# docker compose logs immich-machine-learning -f
immich_machine_learning  | INFO:     Started server process [1]
immich_machine_learning  | INFO:     Waiting for application startup.
immich_machine_learning  | Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
immich_machine_learning  | The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
immich_machine_learning  | Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
immich_machine_learning  | download_path: /cache/models/buffalo_l
immich_machine_learning  | Downloading /cache/models/buffalo_l.zip from https://github.com/deepinsight/insightface/releases/download/v0.7/buffalo_l.zip...

root@immich-ct:~# docker compose logs immich-machine-learning -f
immich_machine_learning  | INFO:     Started server process [1]
immich_machine_learning  | INFO:     Waiting for application startup.
immich_machine_learning  | Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
immich_machine_learning  | The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
immich_machine_learning  | Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
immich_machine_learning  | download_path: /cache/models/buffalo_l
immich_machine_learning  | Downloading /cache/models/buffalo_l.zip from https://github.com/deepinsight/insightface/releases/download/v0.7/buffalo_l.zip...

jrasm91•2y ago

Looks like machine learning is downloading the models Maybe restart it one more time and try again

Latte macchiatoOP•2y ago

Nope, no change. :/ It also seems to be re-downloading the buffalo model every single startup.

jrasm91•2y ago

Do you have the model cache volume?

Latte macchiatoOP•2y ago

I've copy-pasted the compose from here: https://github.com/immich-app/immich/blob/main/docker/docker-compose.yml

GitHub

immich/docker-compose.yml at main · immich-app/immich

Self-hosted photo and video backup solution directly from your mobile phone. - immich/docker-compose.yml at main · immich-app/immich

Latte macchiatoOP•2y ago

Only modified the external port of the proxy.

Alex Tran•2y ago

It looks like the model hasn’t finished downloaded yet

Latte macchiatoOP•2y ago

It's been stuck at that log for a good hour now. I have 400 mbit internet and according to the Proxmox system monitor the download finishes within under two minutes. Afterwards there's no more network activity.

Alex Tran•2y ago

Can you remove the model_cache volume, then docker-compose pull to get the latest release from last night?

Latte macchiatoOP•2y ago

Done so, same end result. Was already running the latest docker containers btw.

jrasm91•2y ago

Maybe try manually downloading that file and placing it in the models cache folder Then restarting the container

Alex Tran•2y ago

Strange. What is your system spec/os?

Latte macchiatoOP•2y ago

This is a Debian 11 LXC, running on Proxmox. The CPU is a Intel Core i5-1135G7, so standard x86_64.

Alex Tran•2y ago

Last resort, can you try what Jason asked above to download the file@manually?

Latte macchiatoOP•2y ago

That machine learning container is weird. Neither ping, wget or curl are installed, and connections to deb.debian.org seem to fail. I'll try to re-pull it.

jrasm91•2y ago

You aren't using any custom dns or firewall stuff are you?

Latte macchiatoOP•2y ago

The DNS requests go through fine and my OPNsense isn't restrictive. Queries show up in my Technitum server and everything. Also works in the other containers... :/ I'm currently re-pulling the images of the Immich stack. Maybe that will fix it.

jrasm91•2y ago

Definitely seems very strange

Latte macchiatoOP•2y ago

After pulling the latest version it now works absolutely perfectly. I frankly have no idea why, but it works, so...

Cyber_Star•2y ago

I'm having the same problem. Did you just repull everything again? @Latte macchiato

Latte macchiatoOP•2y ago

Yep. I frankly have no idea what actually fixed it. I did do a kernel update a few hours after and did a full node reboot, so maybe that had an impact as well?

Cyber_Star•2y ago

sadly enough this did not fix it for me Also tried removing the models-cache volume and restarting the containers but without success.

Cyber_Star•2y ago

Error Log from microservices:

Cyber_Star•2y ago

Left it alone and suddenly the icons of the people appear and it seems to be working?

Alex Tran•2y ago

lol Can you check your machine learning container?

Cyber_Star•2y ago

those are the most recent logs

Cyber_Star•2y ago

These are still the most recent logs of the microservices:

Alex Tran•2y ago

If you want to make sure, run recognize faces job for missing assets

Cyber_Star•2y ago

If I do that nothing happens in the logs. But for a second i can see this

Cyber_Star•2y ago

after a second it goes back to 0 0

Cyber_Star•2y ago

But i can see faces

Alex Tran•2y ago

yeah that one active job is for queuing assets with missing faces so apparently it runs on all faces already

Cyber_Star•2y ago

Strange that it suddenly fixed itself 🙂

Alex Tran•2y ago

self-healing 😛 How many assets do you have? if you want to make sure you can run the recognize faces job on all assets again

Cyber_Star•2y ago

Ran it on all assets again and no errors anymore XD

Alex Tran•2y ago

great, you don't have that many assets so it should be quick

Karaguya•2y ago

Hello everyone, appearently I have the same problem with machine learning. I get ECONNREFUSED errors in my microservices log: I already tried to restart the stack and installed immich using docker compose with the latest .yml and .env from github My machine learning log locks like that:

Karaguya•2y ago

Alex Tran•2y ago

Did you perform docker-compose down then up?

sogan•2y ago

hmm, this issue seems a bit different than the above ones since here it's trying to reach port 3002 while ml is on 3003.

Karaguya•2y ago

I performed docker compose down and up. While doing that I got typsense errors (No API key found). I guess this is because I added the line manually in portainer as environmental variable but it seems like the "=" then is missing in the .env. After adding and saving it, docker compose up ran without typesense errors but now I get following when I want to use a machine learning job like facial recognition

Karaguya•2y ago

After deleting the model cache and recreating the machine learning container I get the ECONNREFUSED error again in the Microservices log Ok, I'm a bit ashamed but I seems like that in my .env the port for the machine learning url was set on 3002. After changing it to 3003 the facial recognition worked and I get no errors anymore 🙂

sogan•2y ago

lol sometimes it's the tiniest things. glad to hear it works now!

Gaming

Programming

Facial Recognition ECONNREFUSED

Did you find this page helpful?