Hub cache or other solution for "no outgoing internet access" server
I am planning on using Crowdsec on some servers that are not able to fetch data from the internet. What are my options if I still want to keep the collections, parsers, scenarios, ... up2date? AFAIK there is no hub caching or similar in Crowdsec, so I was thinking about one-way syncing the hub dir from a server that is able to connect to the internet and regularly pulls the updates. I'm not a huge fan of these hacky solutions, so if anyone has a better idea, please help me out.
6 Replies
Important Information
Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command
/resolve or press the green resolve button below.Log Files
If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.
Guide Followed (CrowdSec Official)
If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.
Screenshots
Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.
© Created By WhyAydan for CrowdSec ❤️
Forgot to mention: IMHO the best solution would be to not only have a
hub_branch option to define which branch to pull from the official Crowdsec Github repo but also have a hub_repo_url option to define which repo to pull from. That way I could simply clone that repo on a dedicated internal git server and have my CS servers pull the hub updates from there.Crowdsec does not directly interact with the hub git repository, but users an API hosted by us to get the content of everything.
You should be able to copy the content of the hub repository to
/etc/crowdsec/hub/ , and then cscli should not attempt to download anything from the internet.
You can also manually download the content of the index file from here https://cdn-hub.crowdsec.net/crowdsecurity/master/.index.json?with_content=true
The index file will contain the content of every item, and crowdsec will get whatever it needs directly from it
But there are a few things to be aware of:
- Some scenarios/parsers reference datafiles which are not stored in the hub: they can be in a git repository, a S3 bucket or even download resources directly from 3rd parties (eg, cloudflare IPs list for whitelists are downloaded directly from cloudflare)
- There's a cron that will try to automatically call cscli hub update/upgrade every day, so you might also want to disable itThank you, that's a nice solution and less hacky than anticipated. Love your work guys!
@blotus I managed to get back to that solution but I couldn't get it to work since "cscli hub upgrade" will still require internet because it wants to run a version check.
Here's the version check. I will test if the version check is omitted if I set the hub branch via config (my go knowledge is a bit rusty)
https://github.com/crowdsecurity/crowdsec/blob/master/cmd%2Fcrowdsec-cli%2Frequire%2Fbranch.go#L91
GitHub
crowdsec/cmd/crowdsec-cli/require/branch.go at master · crowdsecur...
CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI. - crowdsecurity/crowdsec
Yes, that worked.