Immich becomes temporarily unresponsive after watching several videos, with Cloudflare Tunnel
I have noticed a weird issue on the web interface: sometimes the videos will not load properly. This happens after watching several videos in a succession (each loading pretty much instantly). When it starts, no video will load (it will just show a spinner), neither any new one, nor any previously watched one. If I wait patiently, the videos will start working again, just to break again later (after watching another series).
While in this unresponsive state, the network tab shows that the requests are simply taking a very long time (e.g. > 30 seconds). While in this state, the server does not respond at all, so I cannot open immich in a new tab or in an entirely different browser. It goes away after a minute or so.
It seems to be related somehow to me using a Cloudflare Tunnel, as indeed I don't have the issue at all if I eliminate Cloudflare from the equation (if I add an entry in my hosts file that points my domain to the private IP from the LAN).
My config: I am running a home server with an 11th gen intel i5 (Linux), with nvme SSD, 16 GB RAM, gigabit internet connection (both on the server and on the client). My Immich instance is accessible through the internet by Cloudflare ZeroTrust (free tier). I don't see any apparent performance issue on the server (~0% CPU usage while it happens). I also have some other apps (e.g. Nextcloud), and I have an nginx set up as reverse proxy (with HTTPS), and with subdomains for each app. Instead of having to open a port to forward to nginx on my router, and somehow having to update the IP address on my domain (and sub-domains) whenever this changes (I have dynamic IP), I have free-tier Cloudflare Zerotrusta account, with a Tunnel set up (the home server has cloudflared running as well), and on the tunnel I have a public hostname set for each of my applications (each with its own subdomain, and with a proper HTTP Host header set)
26 Replies
From the main channel, do the cloudflared logs say anything?
Do you go through the cloudflare tunnel while you are at home as well? Or just while you are remote?
It sounds like you are running into some type of limit that is dropping off after a period of time. Like a concurrent connection limit or bandwidth limits, which becomes available again after a period of time.
I think that cloudflare is limiting traffic for videos.
The cloudflared logs show several messages such as these:
But I seem to get these while using the app - I was walking through my video collection, hovering through the entries, and the network tab showed some
NS_BINDING_ABORTED entries, which I thin correspond to cancelled transfers
yes, I do use it while at home as well - I just use the domain name, which points to cloudflare
this looks like some kind of rate limiting to me as well, the more I think about it. I also noticed what while Immich is "unresponsive", I cannot reach any of the other apps either that I have running alongside Immich, such as Nextcloud
but I'm not sure if it is specific to videos, as, after all, all the traffic between my browser and nginx is encrypted (through https), as I am using nginx with Letsencrypt certificates
@thomas I don't know whether it's establishing the TCP connection that gets stalled is what happens here; how could I check that? maybe with Wireshark?Is there any reason why you don't run split dns and bypass the unnecessary round trip while on your lan?
nope, I just didn't know that was an option. Can you maybe give me some link to the documentation or article on the subject?
do I need to run the WARP client on my computer for that?
Does your public dns for your domain/sub domains point to a cloudflare ip?
Depending on your setup it may or may not require a lot of effort to setup, but basically you run a local dns server which has similar A name records for your public domains, but point to internal ips. It forwards all the rest of the requests to an upstream dns, which can resolve everything else.
If a request to your domain happens within your network it gets resolved to a private ip via a private dns server. If a request to a domain happens outside of your LAN it'll get resolved with the public ip and route through cloudflare back into your network as usual.
Kind of, my public DNS (immich.cozycozy.space) is set up as a CNAME with the value of
63fccf79-aa9b-4fd2-9546-fe3eecf956dc.cfargotunnel.com - all this was set up for me automatically by me adding a "public hostname" rule under the tunnel in Cloudflare Zero TrustYup. This is public DNS record to route traffic for
immich.cozycozy.space to cloudflare.
If you think about it, it is quite funny imo. You have an open connection from your phone/computer that gets routed to your router, to your isp, then to cloudflare, then back to your isp, then back to your router, then back to your local server.
Unrelated to all that, you may still run into the same problem when watching videos remotely.I think I understand. I could run this DNS sever on my home server (or maybe on a separate device, such as a raspberry pi), and I could just configure my router to use that address as the default DNS server through DHCP... it makes perfect sense, and it doesn't need anything cloudflare-specific
I presume you’re using the latest version of cloudflared?
Yeah. It can be complicated though. Some routers don't allow being configured to use local dns servers. It's considered a security issue (some malware changes the dns to an internal one and intercepts all your traffic or something).
Can you take a screenshot of the network tab from your browser?
In that case you have to use a different dhcp client which doesn't have that restriction.
almost the latest. My server runs Arch Linux, and the AUR package https://aur.archlinux.org/packages/cloudflared-bin is on the penultimate version only (2023.7.2, while the latest is 2023.7.3). Do you think it's related?
I don’t know, I’d have to read the release notes
I see, but my router actually does support it. Will definitely do this, it makes perfect sense, and it's the perfect excuse for me to set up my own DNS server as well.
I was hovering through the videos, loading them one-by-one, but not watching them through. The red lines correspond to requests for videos that did't load - were made while in this "unresponsive" state. Notice the penultimate request on the bottom that was answered in ~46 seconds. Then it became responsive again (and thus the last request went through fast).
What browser is this? Firefox? Does it happen in chrome?
Yes, this is Firefox, and the same thing happens in Chrome as well. I can make a similar screenshot over there also if you'd like.
@jrasm91 thanks for your explanations on the split DNS!
no its fine jw
time to first byte, weird
and cloudflared points directly at immich, not nginx?
Cloudflare points to nginx, because I am relying on nginx setting up HTTPS for me (with letsencrypt certificates)
btw, I just opened an issue on cloudflare's community forums: https://community.cloudflare.com/t/temporary-network-issues-with-zero-trust-tunnel/543505
Cloudflare Community
Temporary network issues with Zero Trust Tunnel
I am self-hosting a photo and video gallery application Immich (https://immich.app), and I experience network to my service getting temporarily “blocked” if I go peek through my video collection and generate some traffic. The “blockage” goes away after about 30-50 seconds. The application is hosted on a machine with nginx reverse proxy (with ht...
I am aware that there might be a way to rely on Cloudflare for TLS/HTTPS as well, but I am not sure if I want to go that way (privacy)
nginx is probably not the culprit here because I don't have the issue if I bypass cloudflare and directly across nginx over the private network (I tested this by a hosts file entry, setting the domain name to the private ip)
I was more just wondering if multiple reverse proxies was causing something strange to happen