I
Immich•2y ago
rbapo

immich_machine_learning failing due to TLS error

Hey folks, I'm trying to move my immich_machine_learning to a hosted instance in oracle cloud. When main.py starts running it throws the following error: requests.exceptions.SSLError: 
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /microsoft/resnet-50/resolve/main/config.json (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1002)')))
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /microsoft/resnet-50/resolve/main/config.json (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1002)')))
Connecting to the docker via /bin/bash and trying to run apt update (just as an example to test https - returns the following): 
root@6cc5f1456777:/usr/src/app# apt update
Err:1 https://deb.debian.org/debian bullseye InRelease
  Could not handshake: A TLS fatal alert has been received. [IP: 199.232.82.132 443]
root@6cc5f1456777:/usr/src/app# apt update
Err:1 https://deb.debian.org/debian bullseye InRelease
  Could not handshake: A TLS fatal alert has been received. [IP: 199.232.82.132 443]
The cert store on the image does seem to be populated. I thought this might be a problem with the image, so I grabbed the nginx image and tried running apt update on it - that one completed without any issues. My docker compose: 
version: "3.8"

services:
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

volumes:
  model-cache:
version: "3.8"

services:
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

volumes:
  model-cache:
Any help would be appreciated 🙂
12 Replies
rbapo
rbapoOP•2y ago
If anyone has any input on this one it would be great... looking for someone that set up the ML container on a remote host ideally..
rbapo
rbapoOP•2y ago
This one seems pretty related: https://github.com/huggingface/transformers/issues/17611 But I'm not super sure - I tried editing the container and adding the CURL_CA_BUNDLE env variable - but it didn't seem to have a positive effect
GitHub
SSLError: HTTPSConnectionPool(host='huggingface.co', port=443) · Is...
I'm trying in python: from sentence_transformers import SentenceTransformer sbert_model = SentenceTransformer('all-MiniLM-L6-v2') and I get this error: SSLError: HTTPSConnectionPool(hos...
sogan
sogan•2y ago
does running apt-get install ca-certificates in the container work?
sogan
sogan•2y ago
https://stackoverflow.com/a/37971185
Stack Overflow
SSL certificate verification fails inside docker container on speci...
I'm running into a strange problem with certificates that I can't figure out how to debug. When I run wget inside of a docker container on one specific server it cannot verify certificates. The sam...
rbapo
rbapoOP•2y ago
I checked - it was already installed
sogan
sogan•2y ago
there’s also update-ca-certificates you can try. besides that it’s hard to say. is it just https, or does http not work either?
rbapo
rbapoOP•2y ago
(removed previous message which had incorrect info). This is really interesting/weird. Seems to be failing on any https with this container - same error even with google.com: 
 docker exec -it immich_machine_learning bash
root@e936b2a8b8b9:/usr/src/app#  python3 -c "import requests; print(requests.get('https://www.google.com').text[0:100]);"
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/venv/lib/python3.11/site-packages/urllib3/conne
 docker exec -it immich_machine_learning bash
root@e936b2a8b8b9:/usr/src/app#  python3 -c "import requests; print(requests.get('https://www.google.com').text[0:100]);"
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/venv/lib/python3.11/site-packages/urllib3/conne
What I don't understand is how come I'm the only one that's seeing this 🙂 I'm wondering if anyone would be willing to try the following: 1. docker exec -it immich_machine_learning bash 2. openssl s_client -connect google.com:443 For me this throws an ssl error as well I'm wondering if the reason people aren't seeing this is because the models are already cached for them
sogan
sogan•2y ago
i'm wondering if it has something to do with it being hosted in oracle cloud. can you confirm that it works when running locally?
rbapo
rbapoOP•2y ago
Yup, I just confirmed that it happens with the nginx container too. It’s definitely my vm. Ill update if I am able to fix it 🙂 Thanks for the help so far!
sogan
sogan•2y ago
np! hope you can get it working
rbapo
rbapoOP•2y ago
It looked like a networking issue... so I just changed the network_mode to host. Looks like it's working now 🙂 The instance is completely non-responsive - looks like it's doing ML 🙂
sogan
sogan•2y ago
awesome, glad to hear it

Did you find this page helpful?