Immich•4mo ago

Remote Machine Learning - Load Balancing

Is it possible to setup load balancing for remote machine learning under the below scenario: - Immich running on Synology DS220+ on docker - Tailscale access to remote server with both an i5 and Graphics to support both CPU and iGPU processing of assets Can anyone advise how I could set this up? The remote docker containers are both running properly for jobs when checked. I’m only trying to figure out the load balancing portion.

10 Replies

Immich•4mo ago

:wave: Hey @Vat.Kris, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

Immich•4mo ago

Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy). 6. :blue_square: uploaded the relevant information (see below). 7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.

GitHub

immich-app immich · Discussions

Explore the GitHub Discussions forum for immich-app immich. Discuss code, ask questions & collaborate with the developer community.

FAQ | Immich

User

GitHub

Issues · immich-app/immich

High performance self-hosted photo and video management solution. - Issues · immich-app/immich

Athul•4mo ago

which 2 ML servers are you trying to balance b/w? I thought there's only one remote ml instance.

Vat.KrisOP•4mo ago

Well the docs say there can be multiple. For my setup I’ve got remote ml on CPU, another remote container with hwaccel for GPU, and I also want to utilise my NAS GPU for hwaccel. I’ve got a ton of assets (2TiB).

Tempest•4mo ago

I added this to my docker conf

  nginx:
    container_name: immich_load_balancer
    image: nginx:alpine  # Use the official Nginx image
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf  # Mount your custom Nginx config
    restart: always
    env_file:
      - .env
    depends_on:
      - immich-machine-learning
    healthcheck:
      test: ["CMD", "nginx", "-t"]  # Check the Nginx configuration
      interval: 30s
      timeout: 5s
      retries: 3

  nginx:
    container_name: immich_load_balancer
    image: nginx:alpine  # Use the official Nginx image
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf  # Mount your custom Nginx config
    restart: always
    env_file:
      - .env
    depends_on:
      - immich-machine-learning
    healthcheck:
      test: ["CMD", "nginx", "-t"]  # Check the Nginx configuration
      interval: 30s
      timeout: 5s
      retries: 3

and this as my .conf file

http {
  
    # command to reload conf
    # docker exec -it immich_load_balancer nginx -s reload

    # set large timeouts
    proxy_read_timeout 600s;  # 10 minutes
    client_max_body_size 10000M;  # Set to a large value (100 MB)
    client_body_buffer_size 10M;  # Set buffer size to 10 MB (in memory) so we're not constantly writing to disk

    # Log format (uncomment if needed)
    log_format main '$time_local $upstream_addr $status $upstream_status';

    # Output access logs to stdout
    access_log /dev/stdout main;

    # Output error logs to stderr and a specific file
    error_log /dev/stderr warn;  # Logs to stderr

    upstream backends {
        server immich_machine_learning:3003 max_conns=3 max_fails=1 fail_timeout=5m;
        server 192.168.1.3:3003 max_conns=3 max_fails=1 fail_timeout=5m;
        server 192.168.1.4:3003 max_conns=3 max_fails=1 fail_timeout=5m;
        
        server immich_machine_learning:3003 backup;
    }

    server {
        listen 3003;

        location / {
            proxy_pass http://backends;  # Remove trailing slash to prevent issues

            # Handle specific HTTP error codes
            proxy_intercept_errors on;  # Enable error interception
            error_page 400 401 403 404 405 408 500 502 503 504 = @fallback;  # Redirect for errors
        }

        # Fallback location for handling errors
        location @fallback {
            proxy_pass http://immich_machine_learning:3003;  # Redirect to the backup server
            #proxy_pass http://192.168.3.165:3003;
        }
    }
}

events {}

http {
  
    # command to reload conf
    # docker exec -it immich_load_balancer nginx -s reload

    # set large timeouts
    proxy_read_timeout 600s;  # 10 minutes
    client_max_body_size 10000M;  # Set to a large value (100 MB)
    client_body_buffer_size 10M;  # Set buffer size to 10 MB (in memory) so we're not constantly writing to disk

    # Log format (uncomment if needed)
    log_format main '$time_local $upstream_addr $status $upstream_status';

    # Output access logs to stdout
    access_log /dev/stdout main;

    # Output error logs to stderr and a specific file
    error_log /dev/stderr warn;  # Logs to stderr

    upstream backends {
        server immich_machine_learning:3003 max_conns=3 max_fails=1 fail_timeout=5m;
        server 192.168.1.3:3003 max_conns=3 max_fails=1 fail_timeout=5m;
        server 192.168.1.4:3003 max_conns=3 max_fails=1 fail_timeout=5m;
        
        server immich_machine_learning:3003 backup;
    }

    server {
        listen 3003;

        location / {
            proxy_pass http://backends;  # Remove trailing slash to prevent issues

            # Handle specific HTTP error codes
            proxy_intercept_errors on;  # Enable error interception
            error_page 400 401 403 404 405 408 500 502 503 504 = @fallback;  # Redirect for errors
        }

        # Fallback location for handling errors
        location @fallback {
            proxy_pass http://immich_machine_learning:3003;  # Redirect to the backup server
            #proxy_pass http://192.168.3.165:3003;
        }
    }
}

events {}

you're kind of on your own to build a load balancer. Take notes from that if you wish, but there are many many different ways of accomplishing the same

Mraedis•4mo ago

You can just yeet as many links in the ML list as you like, it'll always take the first one that responds So it's a bit of a dumb load balancer, but simple works

Athul•4mo ago

I don't think the kind of load balancing you're looking for can be done. However, you could just use the most powerful device for the ml stuff. I mean, the cpu is not needed for ml if you've already got a GPU. And in general, just go with the beefiest hardware. Load balancing is only needed if your devices are roughly equal in capability. And like Mraedis said, you can add many ML endpoints and immich chooses the top-most available one.

Tempest•4mo ago

I did load balancing with my script. Beefy hardware gets more requests, less beefy hardware gets less requests. I even had a secondary script that watched containers die (back when openvino had issues), and rebooted them and provided a requests/second per container. It certainly is possible, it's just dependant on the effort you want to put in

Vat.KrisOP•4mo ago

So what I gather is that it’s possible, has a bit of effort to be put in, and may not all be worth it in the end. I’m going to try it anyway (because I’m stupid like that!), and I like shaving seconds off a task 😂 And just fyi here is the relevant section from the docs: Among other things, it may enable the use of different APIs on the same server by running multiple containers with different configurations. For example, one might run an OpenVINO container in addition to a CUDA container, or run a standard release container to maximize both CPU and GPU utilization.

Immich•4mo ago

This thread has been closed. To re-open, use the button below.

Gaming

Programming

Remote Machine Learning - Load Balancing

Did you find this page helpful?