Sudden 522 errors on all domains
Shortly after I went to bed two nights ago, all my domains started getting 522 errors. As far as I have been able to determine, no configuration on my end changed. Nginx Proxy Manager is still serving certs and can see my servers, gateway still has CF IPs whitelisted and 80 and 443 are forwarded to NPM, & I verified with my ISP that ports 80 & 443 are not blocked. Public IP is dynamic, but it has not changed. CF configuration doesn't appear to have changed either. Just in case, I have ripped and rebuilt a lot of the above, just in case, and rebooted everything at least once. But so far no dice. Was wondering if anyone here has any insight.
36 Replies
522 is Cloudflare cannot connect to your origin: https://community.cloudflare.com/t/community-tip-fixing-error-522-connection-timed-out/42325
I would check the following:
Check your SSL Mode, in your website in the Cloudflare dashboard under SSL/TLS -> Overview. It should not be Flexible. It should be Full Strict or Strict (SSL-Only). Flexible means connecting over 80/insecure which can be blocked or mitm'd, or cause other issues.
Which PoP do you connect to? If you go to https://yourdomain.com/cdn-cgi/trace, what's the
colo line? SJC-PIG? Could be related to one of CF's rollouts if so, would be helpful for us to track those issues but wouldn't help you fix the issue.
If that's all set, I would then try requesting your origin just like Cloudflare would.
curl --resolve example.com:443:<ip> https://example.com/
replace example.com with your domain
(can replace <ip> with 127.0.0.1 if in shell on same machine as the web server, or remote ip if you don't have a firewall blocking it/or can whitelist yourself)
might need -k if cert isn't trusted
If it works locally, try remotely and whitelisting your IP.
It sounds like you're self-hosting, a true test of connectivity would need to be from a remote server if you have any to test on. Curl overriding to your own IP but from the same network would just get hairpinned and not be the same as Cloudflare connecting in. It's happened before that some ISPs mess with 80/443, even if they say they aren't blocked.
If you just wanted a quick fix, tunnels would likely fix it, are great for dynamic IP situations/completely free/bypass any NAT/and don't expose your IP at all or require you to open any ports: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/Cloudflare Docs
Create a remotely-managed tunnel (dashboard) · Cloudflare Zero Trus...
Follow this step-by-step guide to get your first tunnel up and running using Zero Trust.
I am self-hosting. And I'm open to CF tunnels as an option, but even if I switch to those, I'd like to figure this issue out, even if just to have learned how.
- All domains are set to Full Strict
- When I run the curl command using 127.0.0.1 from within the Nginx Proxy Manager's docker container, I get the expected output. But when I run it from the host the docker is on or any other device on the same network, I get the 522 error.
So it would seem that something borked in my stack overnight if I get a 522 locally. That is really odd.
But when I run it from the host the docker is on or any other device on the same network, I get the 522 error.On the host, are you overriding it properly? Docker isn't my expertise but I believe on host you'd want to override it to the machine's public ip so it routes properly, or potentially the container name. It sounds kinda like when you get the 522 you're failing to override and going out to Cloudflare, what do the response headers look like?
OK, so here's the sanitized commands I'm running:
From within the docker container:
Which returns the contents of the page as expected.
From the docker's host:
and
From another host on same network:
Which all return
error code: 522% as the entire response. Reason for port being 10002 on those is that's what the docker container has 443 exposed on.
OK, I must be formating commands wrong. Ran them verbose and they're definitely sitll calling out to a CF IP.You need to specify the port in the url if it's not standard, that's why I was asking about response headers lol
ex:
curl --resolve domain.tld:10002:HOST_IP https://domain.tld:10002/Ah ah! Learn new things every day. OK, adding the non-standard port to the url results in the expected return: contents of the web page. So I can now curl the contents of the page from the reverse proxy docker, the docker's host, and another machine on the same network.
That's good. Anything about this?
Which PoP do you connect to? If you go to https://<yourdomain.extension>/cdn-cgi/trace, what's the colo line? SJC-PIG? Could be related to one of CF's rollouts if so, would be helpful for us to track those issues but wouldn't help you fix the issue.If you could share the URL of the page behind CF it could be helpful as well
colo=DFW
And by URL of the page, you mean the one that I'm running the trace on?yup
looseaffiliations.pubyup that def 522's. Do you have anything you could test on outside of your network with the same curl resolve override, like how CF is connecting in?
Yes, I have a VPS. So, basically same command, but instead of a local IP, I'd sub in my public IP and use the standard 443 port since that's what it's hitting my router as?
yup
just to clarify though, your setup is like this:
CF -> PublicIP:443 -> Router -> Port forwarding publicip:443:privateIP:10002 -> Host -> DockerContainer 10002:443?
Yes. And that worked flawlessly for the last like two years, then just...stopped, lol. Middle of the night, no updates or anything ran. I did notice something jacked with my nginx proxy manager, where it had lost all the certs, but I've rebuilt those and when I check the certs on the browser when going to the pages, they are the correct new ones.
OK, running
never connects and gets a timed out message.
Even if it lost certs, shouldn't just time out
hmm, try within your home/same network w/ public ip? It should hairpin and turn around, if it doesn't might expose some issue with port forwarding config
Running the exact same command locally fails almost instantly with a no route to host.
CGNAT, maybe?
Sure your IP didn't change?
I am certain. The IP I have in CF is the same one I get by every method I use to check. It sure behaves like it changed tho, lol. But I have a heartbeat monitor on my VPS to notify me if my internet drops and that hasn't gone off either, and it would if IP changed.
And as far as I know (and have been told when I ask) there's no CGNAT or blocking of ports going on on the ISP side. But I'm residential so it's possible the support I get is wrong. I also passthrough the ISP through to my own equipment. And based on what I know (which could be wrong) my public IP address does not indicate I have CGNAT. It's a proper public IP.
Just curl'd my public IP again and it definitely matches what is in CF and has been in CF for the past several months since the last time it changed on me.
so you have double nat and double port forwarding?
Full disclosure: I'm a little fuzzy on what double nat is.
Basically, I've set my ISP's gateway to IP passthrough to my own gateway, which then does the port forward to my reverse proxy.
Full disclosure: I'm a little fuzzy on what double nat is.In essence, your own router's IP is a Private IP from ISP Router and not a Public IP, because the first router/isp equipment is also NAT enabled, so you have two layers handing out private IPs and doing NAT
Gotcha, then yes. And that's how it's been and worked for months now. So something had to have updated/changed somewhere.
yea, was just trying to understand the setup a bit. Ultimately CF can't connect back to your proxy and neither can your VPS, so it's clear the issue is somewhere in that, just weird for it to change all of a sudden and break
I would trust your own router and blame ISP for changing something. Updating/changing router settings, etc.
Yeah, that's what's got me confused. But based on the t/s you helped me with, that seems to be the case. I'll dig into the settings on both / try to get my ISP to fess up if they did something.
"Worst case" I go to tunnels, which would be a new thing to learn, lol.
If I have any further questions as I delve in I'll post here, but I greatly appreciate your help. Got some new stuff to add to my kb.
Tunnels are cool. They connect outbound to Cloudflare and proxy requests in, just like how you connect out to discord.com or any website. No portforwarding, doesn't matter if your IP changes every 10 seconds, no firewall exclusions, etc. Good for home use, but they do tie you into the ecosystem
Well, my DNS and domains are already with CF, lol.
It's a sort of proxy layer on its own, you specify how incoming requests for each hostname are handled, and the tunnel daemon/service you install on one of your systems connects locally to the specified origin
Not that I'm looking for a full how-to here, but can it handle multiple domains and subdomains, that are hosted on multiple different VMs on my stack? Like am I going to have to have an agent running on each VM kinda thing?
can just throw it on the host and have it connect to the container locally. Since it's local traffic same machine/trusted network, you can forgo certificates/encryption too, since user -> encrypted -> cf -> encrypted -> tunnel (running in your network) -> unencrypted, same machine -> origin
, but can it handle multiple domains and subdomains, that are hosted on multiple different VMs on my stack?Yes
Like am I going to have to have an agent running on each VM kinda thing?You can do it eitherway. You could have one tunnel and one cloudflared/daemon per machine, or just set it up on a single machine and have it connect locally to the rest The only thing I would mention is that each tunnel has to be able to connect to all specified origins, so you could do either one tunnel and daemon on a core machine, and the origins are configured with their private IPs and it connects out to the other machines, or do it one tunnel per machine connecting to the service on it via localhost/locally, but you can't install a ton of replicas/daemons for the same tunnel on more then one machine, unless it can reach all of the origins https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/ is a good getting started guide
OK, noted, thank you. And one final clarification: you say you can forgo certificates and such, but I assume there's still a way to make it work with certificates so other people going to a website of mine don't have their browsers freak out?
Awesome, I'll follow up on that guide to get started, thanks!
(and when I say machine, I mean per VM, or however you want to do it)
When I say you can forgo certificates, I mean you can forgo them locally, because:
User -> Cloudflare = uses CF Edge Cert
Cloudflare -> Tunnel Daemon = Encrypted using CF Internal Certs
Tunnel Daemon is on your internal network, so it can connect to services on the same machine or local network insecurely/without certs, without worry of MITM (unless you're worried about someone jacking into your local network lol)
OK, that makes more sense. Thank you.
So Users would still see the lock and everything, it's just locally it would connect, probably like your nginx proxy manager already does, insecurely. Or you can connect securely for fun too, tunnel daemon is just like a simple proxy/nginx proxy manager in that way, can do insecurely or securely
Well, I figured out the issue. I overlooked one specific thing in my gateway...the passthrough from ISP gateway to my own borked and my gateway was pulling the local IP, not the WAN IP.
Sorry for having you do all that t/s with me, but I did learn a lot to carry forward with me.
Basically, all the passthrough settings were correct, the gateway had just lost the WAN IP. I had to reboot the devices in just the right order to get it back.
ahh ok, well at least it was something simple, double nat/routers will do confusing stuff like that sometimes
Yeah, that'll be the first thing I check if this happens again, that's for sure.
And also gives me more incentive to look into tunnels, as that wouldn't have been an issue.