Immich•3w ago

Python error on machine learning pod

On k3s running on a pi4 cluster, using Helm (templated). When I try to spin up v1.131.3 the machine learning pod throws the following error:

$ kubectl logs -n immich immich-machine-learning-d74bb5d4-dknhq
...
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='tcp://10.43.86.160:2283', input_type=str]

$ kubectl logs -n immich immich-machine-learning-d74bb5d4-dknhq
...
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='tcp://10.43.86.160:2283', input_type=str]

If I manually specify the IMMICH_PORT env variable the error isn't thrown, but the pod never becomes healthy:

$ kubectl get events -n immich
...
4m8s        Warning   Unhealthy          pod/immich-machine-learning-5b8c549767-j4h4f    Startup probe failed: Get "http://10.42.4.195:3003/ping": dial tcp 10.42.4.195:3003: connect: connection refused
2m4s        Normal    SuccessfulCreate   replicaset/immich-machine-learning-5b8c549767   Created pod: immich-machine-learning-5b8c549767-2pht5

$ kubectl get events -n immich
...
4m8s        Warning   Unhealthy          pod/immich-machine-learning-5b8c549767-j4h4f    Startup probe failed: Get "http://10.42.4.195:3003/ping": dial tcp 10.42.4.195:3003: connect: connection refused
2m4s        Normal    SuccessfulCreate   replicaset/immich-machine-learning-5b8c549767   Created pod: immich-machine-learning-5b8c549767-2pht5

I'll post more info below as i run into the limit here... Any suggestions are greatly appreciated!

12 Replies

Immich•3w ago

:wave: Hey @Tinkerhell, Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:. References - Container Logs: docker compose logs docs - Container Status: docker ps -a docs - Reverse Proxy: https://immich.app/docs/administration/reverse-proxy - Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA Checklist I have... 1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time). 2. :ballot_box_with_check: read applicable release notes. 3. :ballot_box_with_check: reviewed the FAQs for known issues. 4. :ballot_box_with_check: reviewed Github for known issues. 5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy). 6. :ballot_box_with_check: uploaded the relevant information (see below). 7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable (an item can be marked as "complete" by reacting with the appropriate number) Information In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider: - Your docker-compose.yml and .env files. - Logs from all the containers and their status (see above). - All the troubleshooting steps you've tried so far. - Any recent changes you've made to Immich or your system. - Details about your system (both software/OS and hardware). - Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h). - The version of the Immich server, mobile app, and other relevant pieces. - Any other information that you think might be relevant. Please paste files and logs with proper code formatting, and especially avoid blurry screenshots. Without the right information we can't work out what the problem is. Help us help you ;) If this ticket can be closed you can use the /close command, and re-open it later if needed.

TinkerhellOP•3w ago

My values files:

## This chart relies on the common library chart from bjw-s
## You can find it at https://github.com/bjw-s/helm-charts/tree/main/charts/library/common
## Refer there for more detail about the supported values

# These entries are shared between all the Immich components

env:
  REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}'
  DB_HOSTNAME: postgresql-rw # "{{ .Release.Name }}-postgresql"
  DB_USERNAME: immich # "{{ .Values.postgresql.global.postgresql.auth.username }}"
  DB_DATABASE_NAME: immich # "{{ .Values.postgresql.global.postgresql.auth.database }}"
  # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
  DB_PASSWORD: password # "{{ .Values.postgresql.global.postgresql.auth.password }}"
  IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'

image:
  tag: v1.131.3
[...]
# Dependencies
redis:
  enabled: true
  architecture: standalone
  auth:
    enabled: false

# Immich components

server:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-server
    pullPolicy: IfNotPresent
  env:
    IMMICH_PORT: 2283
  ingress:
    main:
      enabled: false
      annotations:
        # proxy-body-size is set to 0 to remove the body limit on file uploads
        nginx.ingress.kubernetes.io/proxy-body-size: "0"
      hosts:
        - host: immich.local
          paths:
            - path: "/"
      tls: []

machine-learning:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-machine-learning
    pullPolicy: IfNotPresent
  env:
    TRANSFORMERS_CACHE: /cache
    IMMICH_PORT: 2283 # Tried with and without this
  persistence:
    cache:
      enabled: true
      size: 10Gi
      # Optional: Set this to pvc to avoid downloading the ML models every start.
      #type: emptyDir
      accessMode: ReadWriteMany
      storageClass: rook-ceph-block

## This chart relies on the common library chart from bjw-s
## You can find it at https://github.com/bjw-s/helm-charts/tree/main/charts/library/common
## Refer there for more detail about the supported values

# These entries are shared between all the Immich components

env:
  REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}'
  DB_HOSTNAME: postgresql-rw # "{{ .Release.Name }}-postgresql"
  DB_USERNAME: immich # "{{ .Values.postgresql.global.postgresql.auth.username }}"
  DB_DATABASE_NAME: immich # "{{ .Values.postgresql.global.postgresql.auth.database }}"
  # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
  DB_PASSWORD: password # "{{ .Values.postgresql.global.postgresql.auth.password }}"
  IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'

image:
  tag: v1.131.3
[...]
# Dependencies
redis:
  enabled: true
  architecture: standalone
  auth:
    enabled: false

# Immich components

server:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-server
    pullPolicy: IfNotPresent
  env:
    IMMICH_PORT: 2283
  ingress:
    main:
      enabled: false
      annotations:
        # proxy-body-size is set to 0 to remove the body limit on file uploads
        nginx.ingress.kubernetes.io/proxy-body-size: "0"
      hosts:
        - host: immich.local
          paths:
            - path: "/"
      tls: []

machine-learning:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-machine-learning
    pullPolicy: IfNotPresent
  env:
    TRANSFORMERS_CACHE: /cache
    IMMICH_PORT: 2283 # Tried with and without this
  persistence:
    cache:
      enabled: true
      size: 10Gi
      # Optional: Set this to pvc to avoid downloading the ML models every start.
      #type: emptyDir
      accessMode: ReadWriteMany
      storageClass: rook-ceph-block

Immich•3w ago

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

TinkerhellOP•2w ago

Complete python error:

$ kubectl logs -n immich immich-machine-learning-d74bb5d4-dknhq
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/immich_ml/__main__.py", line 6, in <module>
    from .config import log, non_prefixed_settings, settings
  File "/usr/src/immich_ml/config.py", line 105, in <module>
    non_prefixed_settings = NonPrefixedSettings()
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pydantic_settings/main.py", line 176, in __init__
    super().__init__(
  File "/opt/venv/lib/python3.11/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for NonPrefixedSettings
immich_port
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='tcp://10.43.86.160:2283', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/int_parsing

$ kubectl logs -n immich immich-machine-learning-d74bb5d4-dknhq
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/immich_ml/__main__.py", line 6, in <module>
    from .config import log, non_prefixed_settings, settings
  File "/usr/src/immich_ml/config.py", line 105, in <module>
    non_prefixed_settings = NonPrefixedSettings()
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/pydantic_settings/main.py", line 176, in __init__
    super().__init__(
  File "/opt/venv/lib/python3.11/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for NonPrefixedSettings
immich_port
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='tcp://10.43.86.160:2283', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/int_parsing

No one? Any maintainers around who can replicate?

Mraedis•2w ago

I think @bo0tzz is the only one running anything kubernetes 👀

TinkerhellOP•2w ago

Thanks so much! Okay, great possibilities for success, then 😅 I'm not sure it's Kubernetes related, though. It could be an architecture thing, since I'm running on Raspberry Pi

bo0tzz•2w ago

I don't see how this would be kubernetes related, but I also don't have any other ideas

TinkerhellOP•2w ago

An rpi specific issue, maybe? Anyone running on that?

Mraedis•2w ago

rpi is somewhat underpowered for immich so not a lot of people use it, no

TinkerhellOP•21h ago

I was afraid of that... I dont have a ton of images, so thought I'd give it a try. It just seems like a strange error because it looks like a problem within the code, since python throws it. It may only be in the arm image so... no chance of any help for an arm system?

Mraedis•21h ago

Well to offer expertise first we need some. 👀 I have no ideas here

sogan•17h ago

The port for ML is normally 3003, not 2283. The specific error you’re getting though is a little weird. It’s trying to cast the host + port to an int, and the only int config there is the port. So somehow IMMICH_PORT is evaluating to the host + port and not just the port Also, IMMICH_MACHINE_LEARNING_URL is deprecated and shouldn’t be used

Gaming

Programming

Python error on machine learning pod

Did you find this page helpful?