"Machine Learning Request.. Failed for all URLs" on new setup
Hi, I'm new to Immich but have managed to get the server up and running on my Synology NAS 920+ (I'm using Container Manager). I also have a PC on the network with an GTX 1080 on which I am running docker with the immich-ml container (which also seems to be running ok). I've gone through the web UI setup and listed the immich-ml container in the Machine Learning section (above the default), but Immich doesn't seem to be using the external GPU/container. I've double-checked the IP address and having included a huggingface code. I've attached the yamls, env and relevant logs (immich-server, immich-machine-learning on the NAS; and immich-ml on the PC). The postgres and redis seem to be running fine. It is probably something simple I've missed as a newcomer and not very techy person. Thanks for any help. [edit] Immich version 1.135.3 [edit2] the issue seems to be with the "Machine Learning Request.. Failed for all URLs" in the immich-server, and the "fetch failed" [edit3] I can access the PC's IP and port (http://192.168.0.xxx:3003/) from another device on local network and see: {"message":"Immich ML"}, so there shouldn't be a firewall issue on the PC (I've also put an allow rule in my NAS firewall).
22 Replies
:wave: Hey @r1k,
Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich :immich:.
References
- Container Logs:
docker compose logs docs
- Container Status: docker ps -a docs
- Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
- Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA
Checklist
I have...
1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
- The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.You're running ML on a remote host? Is their firewall open to allow the incoming request?
Yes, I can also ping the machine from the server. Also the local ml isn’t working either
It's loading the model frequently so requests are coming in
My guess is some funky synology firewall issue
So you mean the immich_server is managing to contact the ML container, but somehow the ML isn't responding properly? The local ML is within the same project and docker network so there shouldn't have any relation to the Synology firewall for at least that aspect. If I curl both http://immich_machine_learning:3003/ and the container's specific IP and port from the immich_server container I get the 200 OK response, so that seems to be consistent. In the Immich GUI settings I have the server address set as: http://immich_machine_learning:3003 so again that all seems to track. I'm not very good at understanding these logs and I'm still a basic docker and networking user so I'm not really sure what else I can do to problem solve this one. I currently have machine learning disabled in the GUI settings. When I had this enabled the CPU usage on the NAS was close to 100% previously. I have just re-enabled the machine learning in the GUI and just restarted the smart search option (face detection is paused) and I am now getting 81% cpu usage and the attached logs for server and internal ML. Does this mean the external ML isn't connecting, but the internal one maybe is working ok?
The external one is likely the one facing connection issues, while the internal one probably has trouble either loading the model or getting it downloaded, both issues seen frequently on synology
Thanks for this. I haven’t been able to find any relevant threads. Would you happen to have any links I could follow on this?
Not immediately, looking for synology by itself should yield a bunch though
You could try turning the firewall off on the Synology, that would root out Synology firewall issues, and a major starting point for troubleshooting
As I'm focused on external ml, I've turned off the NAS firewall and still no luck. The external ml container does respond when I start the ml task (in this case smart search) in the Immich GUI, but then nothing further happens. I have been able to ping between the NAS and the external PC running the ml. I also get the expected message: "Immich ML" when calling up the ml IP and port. I've verified that CUDA is working as expected with an NVIDIA benchmark. So it seems the network is communicating and the ML should be running ok. Any ideas on what could be missing?
@jrasm91 I was thinking of maybe just doing a full reinstall?
Let me read through everything.
It looks like it is receiving the request but it then fails because of 429 (rate limited).
Yeah this is not an immich-server to immich-machine-learning container issue. This is a machine-learning to huggingface communication/configuration issue. Why is it hitting 429 and why do you have a
HUGGING_FACE_TOKEN?For some reason I was having timeouts with the model, so I tried some different things. Eventually I was able to manually import the model into the cache then the ML seemed to be ok (see the August 2nd post with logs) BUT I still wouldn't actually process anything.
So I seem to have solved that rate limited issue, but I still have the issue between the containers somehow.
You don't have any inner container issues afaik, but it tries to download another model. There is a facial recognition model as well as a clip model.
I did a fresh install of the immich server, db and ml on the NAS. This seemed to get the facial scan working (using buffalo_l wasn't too slow, antelope was almost a standstill) using the internal ML container. Still didn't seem to get any love from the external ml container (even after resetting face detection and only allowing the external ml) which doesn't seem to trigger any response. At this stage, I'd be very happy if I can just get the internal ml working (it seems there are plenty of Synology NAS out there with my specs or worse that don't have issues so this should be feasible). The internal ml initialises the smart search model when I do that scan job, but I can't see anything further. I know the ml works ok with the faces now... but as there is no particular ML logging once the model is initialised, I was wondering if I could check the db to see if the smart search scans are being recorded there? It is possible they are and 1. I'm not aware of it, 2. somehow the GUI smart search part is the issue. Do you know where in the db I should look for this?
There is a smart_search table with an embedding field which is there after ML processes it.
Again, this looks more like a connectivity issue with hugging face in general not a problem with anything else.
Can you ping it download the model offline?
you can see progress by changing your log level to
log (I think) and then monitoring the progress of the immich_server container. The ML container will only show logs for loading/unloading the model into memory (or downloading if required).
If you're unable to download the model from the container, and unable to reach the container from another container, perhaps you have modified your firewall rules quite a bit and there is some leftover networking issue between containers?Ok, finally getting somewhere. Note: I am trialing on a collection of only 20 images (some external, some uploaded). I am only using the internal ml container for now (so don't have to worry about networking issues). Face Detection works perfectly with buffalo_l (have redone multiple times). Running the smart search (currently trying: ViT-B-16-SigLIP2__webli) seems to 1. give logs on the ml container showing that the ml is loaded and running (see attached), 2. Gives a countdown in the Immich admin GUI smart search tab (see attached) and importantly, 3. There IS a smart_search table in database with exactly 20 rows (see attached) in the database! However, when I go the user GUI and type any search term (I have tried "cat", "baby", "fireworks", and "city") it processes for a few seconds then just returns all 20 of the photos, whereas only 1 or a few should get returned. So it looks like the smart search job runs ok, but there is some issue with the contextual search lookup itself, or limiting the . Are there any tables or other things I can look into to further diagnose this?
I have tried the smart searches on both Chrome and Firefox with the same results, and from a different computer.
That’s normal; the search sorts by relevance, but doesn’t filter by relevance
OMG, you are right! Thanks.
Is there any way to filter?
There is no way to place a cutoff, no