504's, 525's, can't get past some access issues

I've just migrated a whole bunch of stuff to a new server, and the domain to access them goes through Cloudflare. Behind Cloudflare is an Nginx Proxy Manager instance, which has LE certificates configured for each subdomain. My router is configured appropriately to forward all requests on associated ports to the correct destination.

I am currently seeing a 504 gateway timeout when requesting one subdomain, but not a different one. Both are configured seemingly identically.

My questions are...

Is there some sort of issue with LE certs behind CF? I would expect there may be if requests are passing through Cloudflare.
Is there anything specific required for subdomain configuration? I've tried adding explicit records for the subdomain, as well as using a wildcard, but neither seems to be overly happy.
Is there anything obvious that stands out to someone more experienced than I?

JoshOP•12/16/24, 9:20 PM

Oh, one more question... If I see the Cloudflare specific 504 error, does that mean that the issue did not propagate at all into my network? It does it hijack potential downstream 504's to display the Cloudflare one?

JoshOP•12/16/24, 9:34 PM

that's good to know, at least... i suspect the problem lies behind CF, but i'm also struggling to find evidence of that

JoshOP•12/16/24, 9:34 PM

wait, no, i am seeing some evidence of that... my access logs show the requests coming in, but it's returning 499's

JoshOP•12/16/24, 9:36 PM

although that's perhaps a weirdness in and of itself, nginx proxy manager is responding with 499 to legitimate requests, and 504 to "trace" requests

JoshOP•12/16/24, 9:44 PM

2024/12/16 21:41:30 [error] 647#647: *182945 upstream timed out (110: Operation timed out) while connecting to upstream, client: redacted, server: redacted, request: "GET / HTTP/2.0", upstream: "http://10.0.221.13:9000/", host: "redacted", referrer: "http://10.0.0.20:7818/"

2024/12/16 21:41:30 [error] 647#647: *182945 upstream timed out (110: Operation timed out) while connecting to upstream, client: redacted, server: redacted, request: "GET / HTTP/2.0", upstream: "http://10.0.221.13:9000/", host: "redacted", referrer: "http://10.0.0.20:7818/"

it's as if one server can't talk to another :thonk:

JoshOP•12/16/24, 9:46 PM

ok, the machine running the reverse proxy for some reason can't communicate with the machine i'm proxying to

JJosh wait, no, i am seeing some evidence of that... my access logs show the requests ...

Chaika•12/16/24, 9:59 PM

499 is nginx saying Cloudflare closed the connection because Cloudflare gave up/timed out, is most likely the sequence there

JJosh ok, the machine running the reverse proxy for some reason can't communicate with...

Chaika•12/16/24, 9:59 PM

makes sense

JoshOP•12/16/24, 10:00 PM

the downside is, i've taken it a bit further, i tweaked some stuff in the proxy manager, and i'm now just getting 521's

JoshOP•12/16/24, 10:00 PM

users are reporting i'm down, too :PepeHands:

JoshOP•12/16/24, 10:01 PM

and now i don't see any requests even hitting my reverse proxy, so something is borked on the CF config side of affairs

JJosh the downside is, i've taken it a bit further, i tweaked some stuff in the proxy ...

Chaika•12/16/24, 10:01 PM

make sure your ssl/tls mode is Full (Strict) under SSL/TLS -> Overview

JoshOP•12/16/24, 10:02 PM

oh, i changed that to full, earlier, to test something else

JoshOP•12/16/24, 10:02 PM

just changed it back though, still 521

Chaika•12/16/24, 10:02 PM

ehh Full should still be fine (well, insecure, but wouldn't cause this error), just not Flexible which is sending http/80 to origin

Chaika•12/16/24, 10:02 PM

521 is just a generic CF can't talk to your origin
https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-521-web-server-is-down

Cloudflare Docs

Troubleshooting Cloudflare 5XX errors · Cloudflare Support docs

When troubleshooting most 5XX errors, the correct course of action is to first contact your hosting provider or site administrator to troubleshoot and gather data.

JJosh users are reporting i'm down, too :PepeHands:

Chaika•12/16/24, 10:04 PM

You said you moved stuff to a new server, where you using CF Before?

JoshOP•12/16/24, 10:04 PM

I was, yeah

JoshOP•12/16/24, 10:04 PM

the only perceivable difference is the new server

JoshOP•12/16/24, 10:05 PM

makes me wonder if there's some mechanism to reset some CF SSL cache

Chaika•12/16/24, 10:05 PM

Well a 521 is just a generic failure to connect, so I would check firewalls/anything you have that could be blocking the connection. You could try going through the front door yourself

curl --resolve example.com:443:<ip> https://example.com/
(can replace <ip> with 127.0.0.1 if in shell on same machine as the web server, or remote ip if you don't have a firewall blocking it)
might need -k if cert isn't trusted

JJosh makes me wonder if there's some mechanism to reset some CF SSL cache

Chaika•12/16/24, 10:05 PM

ssl cache?

CChaika ssl cache?

JoshOP•12/16/24, 10:06 PM

some cache between cloudflare and my network, whereby it stores the SSL certs and complains if they change before it's refreshed them

Chaika•12/16/24, 10:06 PM

connections aren't kept alive for long, and 521 is a failure to establish a connection

JJosh some cache between cloudflare and my network, whereby it stores the SSL certs an...

Chaika•12/16/24, 10:06 PM

wouldn't cause that error

JoshOP•12/16/24, 10:07 PM

oh wait, the proxy manager is now down

JoshOP•12/16/24, 10:07 PM

what is happening in my life, how did it all go so wrong

JJosh oh wait, the proxy manager is now down

Chaika•12/16/24, 10:07 PM

that would do it lol

JoshOP•12/16/24, 10:08 PM

the ui has been responsive this whole damned time

JoshOP•12/16/24, 10:09 PM

i know this isn't your thing at all, but if i don't scream here i can't scream anywhere... why tf does this thing not have an ip address

JoshOP•12/16/24, 10:17 PM

ok, the proxy manager is back online, only this time, the thing that wasn't working earlier now is working, and the thing that was working earlier is now getting a bad gateway :thonk:

JoshOP•12/16/24, 10:18 PM

@Chaika do you mind me asking if you have some place you're happy to try hit a website?

JoshOP•12/16/24, 10:18 PM

also, if your DMs are open, i'm sort of uncomfortable posting links to my stuff publicly

JJosh @Chaika do you mind me asking if you have some place you're happy to try hit a w...

Chaika•12/16/24, 10:24 PM

as long as you mean "hit" as in just try requesting it sure

JoshOP•12/16/24, 10:24 PM

yah nothing more

JJosh yah nothing more

Chaika•12/16/24, 10:26 PM

so map "works" and pg 502's, probably returned directly from nginx

JoshOP•12/16/24, 10:40 PM

gotcha, taking a look internally

JoshOP•12/16/24, 10:40 PM

i've a feeling i'm being cached out of the real situation locally, hence asking

JoshOP•12/16/24, 10:44 PM

ok, solved that with the new (old) proxy manager

JoshOP•12/16/24, 10:44 PM

back online for now, i'll try fix everything again in the morning

JoshOP•12/16/24, 10:45 PM

thanks for your help, much appreciated

2024/12/16 21:41:30 [error] 647#647: *182945 upstream timed out (110: Operation timed out) while connecting to upstream, client: redacted, server: redacted, request: "GET / HTTP/2.0", upstream: "http://10.0.221.13:9000/", host: "redacted", referrer: "http://10.0.0.20:7818/"

504's, 525's, can't get past some access issues

Similar Threads

Similar Threads

Similar Threads