Max number of machines a LAPI can handle
Hi.
Just wondering if there is an upper limit on the number of machines a single LAPI can handle? For example, if I'm reselling VPS WordPress hosting to clients I could eventually end up with hundreds of machines. Ideally they'd all connect to a single LAPI so they can use a single shared db. Could it handle that kind of load?
Cheers
8 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
You have 2 things to consider: machines (log processors) and bouncers.
For machines, the most important part is how often you think they will push alerts: if you expect them to push alerts very often, I'd advise using postgres as a DB backend (sqlite works fine, but it has a limitation of only allowing a single write at a time, which can bottleneck very very quickly if you have a lot of LP reporting alerts at the same time).
Bouncers are maybe a little bit more complex: while they don't push anything, they will incur a lot of reads on the database.
Each time the community blocklists (or any list you pull from CAPI) are updated, they will receive the entire content, which will more than likely cause a lot of meory consumption in crowdsec: we have a feature flag that can help with that
chunked_decisions_stream (see here for enabling it https://docs.crowdsec.net/docs/next/configuration/feature_flags/)CrowdSec | CrowdSec
CrowdSec, the open-source & participative IPS
Another thing to keep in mind is that if you share the same LAPI for every WP instance, they will all share the same decisions (ie, if IP A gets banned on an instance, it will be banned on all the others, which may or may not be what you want)
Thanks. Yeah I was planning on using Postgres. I'll check out
chunked_decisions_stream too.
Shared decisions across all instances is what I was actually interested in, since that means bad actors on one site will be blocked / challenged on every site I'm managing.@blotus You should check out SQLite's WAL journal mode. It allows concurrent writes and pretty much eliminates the performance bottleneck you're talking about: https://highperformancesqlite.com/watch/wal-mode
WAL mode - High Performance SQLite
In the video, I explain how using Write-Ahead Logging (WAL) mode in SQLite can improve database performance significantly compared to Rollback mode. WAL mode allows concurrent reading and writing, making it much faster and efficient because it appends changes to a separate log file instead of altering the main database directly. I also mention t...
we are using WAL mode
WAL allows for multiple readers, but still only a single writer
@blotus Ah right, didn't realise that
(IIRC, we automatically enable WAL mode if we detect you are storing the DB on a non-networked FS, and you can also force it in the configuration)