CD
Cloudflare Developers•7d ago
ioweb-gr

Cloudflare reports 520 on a 444 response from server

Hi We have installed a very comprehensive list of nginx rules to block aggressive bots and malicious scanners, and in doing so our server responds with a code 444 "No response" for such requests. This is very common to respond with 444 on malicious requests because it is very light on nginx and also to hide any type of potential response that could assist the malicious users. Cloudflare should in my opinion respond with 444 as well not 520. Any ideas if that's something we can modify or not?
9 Replies
Walshy
Walshy•7d ago
444 is a non-standard response code and 520 is origin returned unknown error any reason to not just do 429 or 400 or something standard?
ioweb-gr
ioweb-grOP•7d ago
well 444 
Stops processing and returns the specified code to a client. The non-standard code 444 closes a connection without sending a response header. 
Stops processing and returns the specified code to a client. The non-standard code 444 closes a connection without sending a response header.
This effectively stops wasting any resources regarding nginx's processing time much faster than anything else that is going to return headers. In case of DDoS it's the best possible choice There have been vulnerabilities in the past which were easily mitigated if you returned 444 so it's a pretty standard response when you don't want to process a request (e.g. a malware scanner) It's very important under SYN flood or HTTP flood attacks for example Generally it's more efficient in terms of resource consumption as no bandwidth is taken, sockets are immediately freed and there's no need to render any response page for example that you might have for a 404 or 403
ioweb-gr
ioweb-grOP•7d ago
Sorry about the SYN Flood it was a poor copy paste. However it's a solid response code preference in multiple scenarios - e.g. Blocking requests methods that should not be served to you server https://www.acunetix.com/blog/web-security-zone/hardening-nginx/#:~:text=Another%20approach%20is%20to%20add,statements%20in%20the%20location%20context - Mapping bad agents 
if ($block_ua) {
        return 444;
}

The blacklist file (example)

map $http_user_agent $block_ua {
        default           0;
        ~*profound        1;
        ~*scrapyproject   1;
        ~*netcrawler      1;
        ~*nmap            1;
    ~*sqlmap      1;
    ~*slowhttptest      1;
    ~*nikto          1;
    ~*jersey      1;
    ~*brandwatch      1;
    ~*magpie-crawler  1;
    ~*mechanize      1;
    ~*python-requests 1;
    ~*redback      1;
}
if ($block_ua) {
        return 444;
}

The blacklist file (example)

map $http_user_agent $block_ua {
        default           0;
        ~*profound        1;
        ~*scrapyproject   1;
        ~*netcrawler      1;
        ~*nmap            1;
    ~*sqlmap      1;
    ~*slowhttptest      1;
    ~*nikto          1;
    ~*jersey      1;
    ~*brandwatch      1;
    ~*magpie-crawler  1;
    ~*mechanize      1;
    ~*python-requests 1;
    ~*redback      1;
}
- Blocking ai bots https://www.linode.com/community/questions/24842/ddos-from-anthropic-ai#:~:text=%E2%80%A6you%20can%20add%20extra%20lines,at%20all%20to%20the%20request - Siliently discarding requests to known attack vectors like wordpress xmlrpc if you're not using it or requests to 
.well-known/.*php files 
.well-known/.*php files
DDoS from Anthropic AI | Linode Questions
Going by Twitter, I'm not the only one who's seeing a huge amount of bot traffic coming from AI company Anthropic. Their bot ("ClaudeBot") doesn't respect robots.txt or ai.txt and they appear t...
Acunetix
Tips for Nginx server hardening
Currently, Nginx is the second most popular web server (based on a study of the top 10,000 websites). It is lightweight, fast, robust, supports the major operating systems and is the web server of choice for Netflix, WordPress.com and other high traffic sites.
ioweb-gr
ioweb-grOP•7d ago
- Protecting against requests without a server name header as a catch-all directive that closes connections to unspecified hosts - It's also used in conjunction with ModSecurity, OWASP rules and fail2ban instead of showing 403 to prevent exposing information.
- Scanning for files existence like sql dumps, archives, sensitive files, .git/HEAD Instead of 404 which can cause them to start brute forcing you discard the connection silently.
- Country blocking 
nginx
geo $block_country {
default 0;
include /etc/nginx/blocked-countries.conf;
}

server {
if ($block_country) {
return 444; # Drop the connection
}
nginx
geo $block_country {
default 0;
include /etc/nginx/blocked-countries.conf;
}

server {
if ($block_country) {
return 444; # Drop the connection
}
This is more efficient than other cases. As to the question why not to prefer 429 in a DoS / DDoS attack? It is because automated tools will adjust timings automatically when seeing 429 or spread through various IPs instead of giving up when receiving "no response". Normal bots will obey the 429 but malware not. In shared hosting not all customers will be behind cloudflare, so you can't block every other IP and allow only cloudflare. Some customers opt to use it and others not. We had cases where Cloudflare was routing traffic through singapore causing a latency of 300+ ms per request visibly slowing down everything so these websites can't use cloudflare. Anyways this is not a debate of whether we should respond with 444 or 429 or something else. The question was if we can alter cloudflare's response in such cases. A simple no it's not possible would be sufficient which evidently is not possible. In that case we'll rely on the cf-connecting-ip header for separating requests that come from cloudflare and other user agents. In that case if it's originating from cloudflare we can respond with 400
semaja2
semaja2•7d ago
Is there a reason traffic is not soley coming from Cloudflare? Any reason for the traffic to bypass Cloudflare? The rules you listed in regards to blocking user agents etc could also be handled by Cloudflare, reducing any potential load on your origin (nginx)
ioweb-gr
ioweb-grOP•7d ago
Hi, yes indeed multiple reasons. 1. This particular case is shared hosting so there are users who don't want to use cloudflare. 2. Plesk doesn't support adding multiple api keys per domain so not everyone wants to give access to their zone in cloudflare. 3. Cloudflare's timeout threshold for a request is too small and it's not configurable so it's a no-go in some cases. Some ecommerce systems need to generate reports on demand which may take up to a few minutes to complete (e.g. to check for stock discrepancies with an ERP or calculate affiliate gross spend ) 4. Single point of management. In order to just block user agents or set up traps you would need to go to each domain and replicate the rules or use an Enterprise account for account level waf rules which still won't apply when dealing with multiple accounts 😦 5. Sometimes it can introduce latency instead of improving delivery. Had a case where traffic was randomly routed through singapore for customers in Doha which led to visible delays in downloading product images for example
morpig
morpig•6d ago
experiencing this issue too but on a different kind of use case than yours. error 520 only occurs when using http/2 between cloudflare <> origin and when the connection is broken, 520 is thrown by cf. solution to this is to disable http2 to origin
asuffield
asuffield•6d ago
it's not possible as stated, because you aren't actually sending a response here. the "status code" 444 is never sent over the network, so nothing can be done with it. if you want a different response then you have to send that response somehow. the recommended way to identify requests coming from cloudflare is to use TLS to origin and check the TLS certificate. you don't have to do it that way, but it's the most robust approach I would also suggest that you do all the content blocking on the cloudflare side via rulesets, and have the origin just refuse to accept connections that don't have cloudflare TLS. trying to do the blocking on the origin side gives you a lot of extra work to do
ioweb-gr
ioweb-grOP•5d ago
the recommended way to identify requests coming from cloudflare is to use TLS to origin and check the TLS certificate. you don't have to do it that way, but it's the most robust approach
That looks like an excellent idea. I believe it can be set up on each nginx server block separately so we can use it per domain . I'll test it out 🙂 Thanks @asuffield this was extremely helpful advice 🙂 We implemented this on all domains behind cloudflare while leaving the rest untouched. Now we can tweak as needed. Thank you.

Did you find this page helpful?