Immich•4w ago

Error connecting to Remote Machine Learning instance running on WSL

Hey! I've been scouring the internet for anyone talking about this issue, but I can't seem to find much on a solution. I have my Immich stack running on my server pc, which doesn't have a dedicated GPU, so I set up a wsl2 docker desktop environment for remote machine learning on my home pc which has a RTX 2080 Super, hoping to leverage some of it's computational power. I currently have my Immich set to use the remote machine learning container, and the local machine learning container as shown in the image. However, no smart search jobs are being processed, and when i check the logs for my immich_server container, I see a bunch of: "Machine learning request to "http://192.168.1.119:3003" failed: fetch failed" Despite this error, I've confirmed that both on my server pc and inside the immich_server container, when I use "curl 192.168.1.119:3003" I recieve "{"message":"Immich ML"}", which would indicate to me that the remote machine learning container is reachable.

13 Replies

Immich•4w ago

:wave: Hey @ElementO, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy). 6. :ballot_box_with_check: uploaded the relevant information (see below). 7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.

ElementOOP•4w ago

- My Remote Machine Learning container is running the latest release of ghcr.io/immich-app/immich-machine-learning:release-cuda - My Immich stack on my server pc uses the following docker-compose (as per recommended on :

name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false
  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:facc1d2c3462975c34e10fccb167bfa92b0e0dbd992fc282c29a61c3243afb11
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:32324a2f41df5de9efe1af166b7008c3f55646f8d0e00d9550c16c9822366b4a
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always
volumes:
  model-cache:

name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false
  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:facc1d2c3462975c34e10fccb167bfa92b0e0dbd992fc282c29a61c3243afb11
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:32324a2f41df5de9efe1af166b7008c3f55646f8d0e00d9550c16c9822366b4a
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always
volumes:
  model-cache:

- My docker-compose for my remote machine learning container:

name: immich_remote_ml
services:
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    extends:
      file: hwaccel.ml.yml
      service: cuda
    volumes:
      - model-cache:/cache
    restart: always
    ports:
      - 3003:3003
volumes:
  model-cache:

name: immich_remote_ml
services:
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    extends:
      file: hwaccel.ml.yml
      service: cuda
    volumes:
      - model-cache:/cache
    restart: always
    ports:
      - 3003:3003
volumes:
  model-cache:

Immich•4w ago

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

Sergey Katsubo•4w ago

Hey @ElementO Have you already checked/eliminated possible typos? As you pointed out

inside the immich_server container, when I use "curl 192.168.1.119:3003" I recieve "{"message":"Immich ML"}", which would indicate to me that the remote machine learning container is reachable.

If so, then it's weird that Immich backend app itself cannot reach this target as well...

ElementOOP•4w ago

Unfortunately yes, I've double checked it everywhere. I initially thought it might've been a problem with my docker desktop setup and windows firewall, but the fact that the remote machine learning container is reachable inside the immich_server container indicated otherwise, and it fully has me stumped. I've been having this problem for a couple months now, and in the process I've updated both the machine learning container and immich server container to their latest images multiple times. I sort of thought it might've been a bug in the server image, as I found some other people on Github and reddit saying similar things about the remote machine learning being finicky. But considering none of those posts were particularly recent, I would've assumed that if it was a bug it would've been noticed and patched already.

Sergey Katsubo•4w ago

I see. Btw, thanks for all details provided and compose configs. (At first glance I thought about Windows quirks, like in https://github.com/immich-app/immich/discussions/20032. But it's not relevant; there is already connectivity successfully tested with curl from outside.) How about a troubleshooting session using traffic sniffer? I'll prepare a short guide soon.

Immich•4w ago

[Discussion] Failed to connect to LAN IP from LAN (Windows, podman) (immich-app/immich#20032)

ElementOOP•4w ago

Curiously, I've just restarted both the immich stack on my server pc and the remote ML container on my home pc, and at the minute I don't seem to be getting the same sorts of errors. Woah thats really weird - I removed the local ML address from the settings in the Immich Dashboard and now it seems to be running just fine? None of the same failed to fetch messages in my immich_server logs at all, and it's seemingly processing smart scan jobs as the number of waiting jobs is decreasing Wow its a miracle, it's also processing my searches in the remote ML container logs 🎉 It seems the issue has solved itself somehow 😂 Thanks for your help! Maybe your general presence persuaded Immich to cooperate 😆

Sergey Katsubo•4w ago

Definitely!

ElementOOP•4w ago

Hmm, just got this in immich_server logs:

[Nest] 7  - 08/17/2025, 11:48:51 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://192.168.1.119:3003" failed: fetch failed

[Nest] 7  - 08/17/2025, 11:48:51 AM   ERROR [Microservices:{"id":"4561bca8-4b63-4321-9ff4-3f34016a72ca"}] Unable to run job handler (SmartSearch): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-16-SigLIP__webli"}}}' failed for all URLs

Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-16-SigLIP__webli"}}}' failed for all URLs

    at MachineLearningRepository.predict (/usr/src/app/server/dist/repositories/machine-learning.repository.js:98:15)

    at async MachineLearningRepository.encodeImage (/usr/src/app/server/dist/repositories/machine-learning.repository.js:116:26)

    at async SmartInfoService.handleEncodeClip (/usr/src/app/server/dist/services/smart-info.service.js:91:27)

    at async JobService.onJobStart (/usr/src/app/server/dist/services/job.service.js:198:28)

    at async EventRepository.onEvent (/usr/src/app/server/dist/repositories/event.repository.js:126:13)

    at async /usr/src/app/server/node_modules/bullmq/dist/cjs/classes/worker.js:496:32

    at async Worker.retryIfFailed (/usr/src/app/server/node_modules/bullmq/dist/cjs/classes/worker.js:774:24)

[Nest] 7  - 08/17/2025, 11:48:51 AM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://192.168.1.119:3003" failed: fetch failed

[Nest] 7  - 08/17/2025, 11:48:51 AM   ERROR [Microservices:{"id":"4561bca8-4b63-4321-9ff4-3f34016a72ca"}] Unable to run job handler (SmartSearch): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-16-SigLIP__webli"}}}' failed for all URLs

Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-16-SigLIP__webli"}}}' failed for all URLs

    at MachineLearningRepository.predict (/usr/src/app/server/dist/repositories/machine-learning.repository.js:98:15)

    at async MachineLearningRepository.encodeImage (/usr/src/app/server/dist/repositories/machine-learning.repository.js:116:26)

    at async SmartInfoService.handleEncodeClip (/usr/src/app/server/dist/services/smart-info.service.js:91:27)

    at async JobService.onJobStart (/usr/src/app/server/dist/services/job.service.js:198:28)

    at async EventRepository.onEvent (/usr/src/app/server/dist/repositories/event.repository.js:126:13)

    at async /usr/src/app/server/node_modules/bullmq/dist/cjs/classes/worker.js:496:32

    at async Worker.retryIfFailed (/usr/src/app/server/node_modules/bullmq/dist/cjs/classes/worker.js:774:24)

But it still seems to be processing jobs for smart search - Sorry I think I've been calling it smart scan this whole time 😂

Sergey Katsubo•4w ago

Intermittent errors like ^^that can happen (e.g. if remote ML is busy). THe jobs will be retried. I think it's fine as long as the number of waiting jobs is decreasing.

ElementOOP•4w ago

Thats great to hear! I think thats problem solved for now then, Thanks!

Immich•4w ago

This thread has been closed. To re-open, use the button below.

Gaming

Programming

Error connecting to Remote Machine Learning instance running on WSL

Did you find this page helpful?