Bouncer decision source across unreliable network
I am running crowdsec with a topology very similar to the multi-server setup described in the docs. It works great!
I have recently added a new machine to my ecosystem that is a VPS VM, outside my local network. It has a log processor reporting back to my LAPI inside my local network and a firewall bouncer getting decisions from that LAPI. It is connected via Wireguard and works fine across this overlay, currently.
However, I've run into an issue with this topology. My local network lost power and all my machines had to shut down, including the LAPI and, of course, the Wireguard peer routing to the VPS. My bouncer was left without an LAPI to query.
This problem was compounded by the fact that I have web services running independently on the VPS that were not affected by this outage.
The obvious solution is to move LAPI to the VPS. But the majority of my traffic is actually forwarded into the local network via TCP proxy. Additionally, the VPS is very low power and I'd prefer to keep LAPI and log processing, for traffic destined for the local network, in the local network so I can use my beefier, local servers.
The second obvious solution is to have a separate LAPI and log processing only on the VPS, only for VPS-bound traffic. But then, the VPS bouncer will only be getting decisions for that traffic and not 99% of actual traffic flowing through it.
My questions/issues with this:
* I don't see an obvious way to run two LAPIs that can sync or are additive in one direction (one LAPI decisions get added to second, which is used as primary LAPI)
* I don't know if I can run two firewall bouncers on the same machine using separate LAPI...will the rules conflict?
* Is there a better way to workaround this topology issue?
Thanks
About multi-server setup | CrowdSec
Introduction
6 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
Here's a simplified example of my current topology
Hey,
- Running 2 different LAPIs and sharing their decisions automatically between them is not supported at the moment. We are working on something for that, but it will be through the console and be part of the enterprise plan.
- What is supported is running 2 (or more) LAPIs connected to the same database (so no sqlite) and sharing the same configuration to have a high availability setup, but it's more complex to implement (you either need to have floating IPs or a proxy/load balancer what will redirect traffic to the LAPI that is still up)
- It won't work, they will both try to manage the same rules/chain/sets, and possibly overwrite each others changes
- You can move only the LAPI to the VPS, and still process the logs locally. Generally, I'd say you want the LAPI to be one your more reliable servers.
Thanks for taking the time to read and the response.
I'll try going with the LAPI in the VPS and see how that does. More than data transfer, though, im concerned about the low (cpu) power of the VPS...
In the past I experienced higher-than-expected cpu load when CS was processing only "moderate" amounts of traffic. https://discord.com/channels/921520481163673640/1341147162037190830/1342347086875852822
I solved this by removing some scenarios, enabling re2, and moving log processing to a beefier machine.
I will try to do some artificial load testing myself, but if you can tell me, what should I expect for LAPI cpu usage when log processors are experiencing high traffic?
depends what you count as high traffic, but LAPI itself should not use much CPU, as it only receives the alerts, and expose the decisions to the bouncers.
So if you have a lot of traffic, but legitimate (ie, does not generate any alert), it will be basically 0.
On the other hand, if the traffic is fully malicious and generate a ton of alerts, then yeah it will likely consume more CPU, but as the offending IPs will be blocked, it should stay manageable
What's the CPU on the VPS ?
OK that makes sense. Most of my traffic is legitimate. Xeon E5-2690 4c/4t @ 2.90ghz with 5GB memory