CrowdSec•10mo ago

NPMplus won’t connect to api

I have an unraid server with a reverse proxy setup with NPMplus, a fork of NPM with a built in bouncer. I have a custom docker network setup and I’ve generated an api key and put it in the crowdsec.conf file within the app data of npmplus. It’s also worth noting I’m using cloudflare proxy with real ip passthrough setup in nginx. The ports are all right from what I can see and so is the api key. Blocked IPs still connect and the bouncer metrics list never updates. Trying to curl from the npmplus container to the crowdsec container gives me an error 401… I see a similar 401 error in the crowdsec container log coming from the correct ip. I’m a bit stumped on what to try next

81 Replies

CrowdSec•10mo ago

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

WillpowerOP•10mo ago

If any logs or config files are needed ping me I haven’t seen any related to api in either npmplus logs or crowdsec logs. I followed the guide on npmplus git and adapted the paths to support unraid and opted to use a docker network instead of host networking I changed the ip in the crowdsec.conf file to the static docker i assigned the crowdsec container

WillpowerOP•10mo ago

WillpowerOP•10mo ago

You can see here that the containers can connect to each other. It just doesn’t work for some reason

iiamloz•10mo ago

This means the api key you using is not valid and is returning a 401 response code because of it.

WillpowerOP•10mo ago

See ive deleted the bouncer multiple times and recreated the api key and it doesn’t make a difference. Ofc i update the api key and reload the npmplus container too And unless I do a curl I don’t have any logs in CrowdSec that shows NPMPlus even tries to connect to CrowdSec’s api It does read logs from docker tho

iiamloz•10mo ago

I just realized, the endpoint you are requesting via curl is not allowed for api key

WillpowerOP•10mo ago

Is there one I can test with?

iiamloz•10mo ago

can you can sending a head request to /v1/decisions/stream curl -X HEAD -H "X-Api-Key: 123456" ......

WillpowerOP•10mo ago

iiamloz•10mo ago

Yeah so that key worked

WillpowerOP•10mo ago

yup, that works so, its not a networking related error docker network wise? let me grab my crowdsec.conf in npmplus

ENABLED=true
API_URL=http://172.18.0.4:8080
API_KEY=sameapikey as curl
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=2500
UPDATE_FREQUENCY=10
# By default internal requests are ignored, such as any path affected by rewrite rule.
# set ENABLE_INTERNAL=true to allow checking on these internal requests.
ENABLE_INTERNAL=false
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=
# Captcha Secret Key
SECRET_KEY=
# Captcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600

APPSEC_URL=http://172.18.0.4:7422
APPSEC_FAILURE_ACTION=deny
APPSEC_CONNECT_TIMEOUT=1000
APPSEC_SEND_TIMEOUT=30000
APPSEC_PROCESS_TIMEOUT=10000
ALWAYS_SEND_TO_APPSEC=false
SSL_VERIFY=true

ENABLED=true
API_URL=http://172.18.0.4:8080
API_KEY=sameapikey as curl
CACHE_EXPIRATION=1
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=2500
UPDATE_FREQUENCY=10
# By default internal requests are ignored, such as any path affected by rewrite rule.
# set ENABLE_INTERNAL=true to allow checking on these internal requests.
ENABLE_INTERNAL=false
# live or stream
MODE=live
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
BAN_TEMPLATE_PATH=/data/etc/crowdsec/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
#valid providers are recaptcha, hcaptcha, turnstile
CAPTCHA_PROVIDER=
# Captcha Secret Key
SECRET_KEY=
# Captcha Site key
SITE_KEY=
CAPTCHA_TEMPLATE_PATH=/data/etc/crowdsec/captcha.html
CAPTCHA_EXPIRATION=3600

APPSEC_URL=http://172.18.0.4:7422
APPSEC_FAILURE_ACTION=deny
APPSEC_CONNECT_TIMEOUT=1000
APPSEC_SEND_TIMEOUT=30000
APPSEC_PROCESS_TIMEOUT=10000
ALWAYS_SEND_TO_APPSEC=false
SSL_VERIFY=true

crowdsec - config.yaml

common:
  daemonize: false
  log_media: stdout
  log_level: debug
  log_dir: /var/log/
config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
  acquisition_path: /etc/crowdsec/acquis.yaml
  acquisition_dir: /etc/crowdsec/acquis.d
  parser_routines: 1
plugin_config:
  user: nobody
  group: nobody
cscli:
  output: human
db_config:
  log_level: info
  type: sqlite
  db_path: /var/lib/crowdsec/data/crowdsec.db
  flush:
    max_items: 5000
    max_age: 7d
  use_wal: false
api:
  client:
    insecure_skip_verify: false
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    log_level: debug
    listen_uri: 0.0.0.0:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    trusted_ips: # IP ranges, or IPs which can have admin API access
      - 127.0.0.1
      - ::1
    online_client: # Central API credentials (to push signals and receive bad IPs)
      credentials_path: /etc/crowdsec//online_api_credentials.yaml
    enable: true
prometheus:
  enabled: true
  level: full
  listen_addr: 0.0.0.0
  listen_port: 6060

common:
  daemonize: false
  log_media: stdout
  log_level: debug
  log_dir: /var/log/
config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
  acquisition_path: /etc/crowdsec/acquis.yaml
  acquisition_dir: /etc/crowdsec/acquis.d
  parser_routines: 1
plugin_config:
  user: nobody
  group: nobody
cscli:
  output: human
db_config:
  log_level: info
  type: sqlite
  db_path: /var/lib/crowdsec/data/crowdsec.db
  flush:
    max_items: 5000
    max_age: 7d
  use_wal: false
api:
  client:
    insecure_skip_verify: false
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    log_level: debug
    listen_uri: 0.0.0.0:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    trusted_ips: # IP ranges, or IPs which can have admin API access
      - 127.0.0.1
      - ::1
    online_client: # Central API credentials (to push signals and receive bad IPs)
      credentials_path: /etc/crowdsec//online_api_credentials.yaml
    enable: true
prometheus:
  enabled: true
  level: full
  listen_addr: 0.0.0.0
  listen_port: 6060

iiamloz•10mo ago

everything I see so far, seems right. The only thing to note your mode is set to live so it only calls the crowdsec LAPI when a request comes into the webserver

WillpowerOP•10mo ago

npmplus.yaml in acquis.d

filenames:
  - /opt/npm/nginx/access.log
labels:
  type: npmplus
---
source: docker
container_name:
 - NPMplus
labels:
  type: npmplus
---
source: docker
container_name:
 - NPMplus
labels:
  type: modsecurity
---
listen_addr: 0.0.0.0:7422
appsec_config: crowdsecurity/appsec-default
name: appsec
source: appsec
labels:
  type: appsec

filenames:
  - /opt/npm/nginx/access.log
labels:
  type: npmplus
---
source: docker
container_name:
 - NPMplus
labels:
  type: npmplus
---
source: docker
container_name:
 - NPMplus
labels:
  type: modsecurity
---
listen_addr: 0.0.0.0:7422
appsec_config: crowdsecurity/appsec-default
name: appsec
source: appsec
labels:
  type: appsec

is stream recommended?

iiamloz•10mo ago

typically we recommend it as it more performant than live (depends on how many RPS your setup gets)

WillpowerOP•10mo ago

ill set that and see if it makes a difference

iiamloz•10mo ago

it will only start pulling from LAPI once a single requests hits the server so make sure to call one of your sites before checking logs

WillpowerOP•10mo ago

any request or one with a detection?

WillpowerOP•10mo ago

no dice

iiamloz•10mo ago

do you have access to npmplus error logs? (nginx error logs)? cause we log to those files

WillpowerOP•10mo ago

something i did when troubleshooting was enable ipv6 for the docker network. does that need to be on? live logs like this? docker logs -f NPMplus

iiamloz•10mo ago

I dont know if npmplus logs all logs to stdout? they might be within the /opt/npm/nginx/

WillpowerOP•10mo ago

iiamloz•10mo ago

and within any of those folders is not an error.log?

WillpowerOP•10mo ago

nope https://github.com/ZoeyVid/NPMplus?tab=readme-ov-file#quick-setup:~:text=Note%3A%20access.log/stream.log%2C%20logrotate%20and%20goaccess%20are%20NOT%20enabled%20by%20default%20bceuase%20of%20GDPR%2C%20you%20can%20enable%20them%20in%20the%20compose.yaml.

GitHub

GitHub - ZoeyVid/NPMplus: Docker container for managing Nginx proxy...

Docker container for managing Nginx proxy hosts with a simple, powerful interface - ZoeyVid/NPMplus

WillpowerOP•10mo ago

I already had LOGROTATE set to true

iiamloz•10mo ago

so from the configuration I see this https://github.com/ZoeyVid/NPMplus/blob/652fd0b94e29dd3b1449fadef7b5a038186019c7/rootfs/usr/local/nginx/conf/nginx.conf#L4

GitHub

NPMplus/rootfs/usr/local/nginx/conf/nginx.conf at 652fd0b94e29dd3b1...

Docker container for managing Nginx proxy hosts with a simple, powerful interface - ZoeyVid/NPMplus

iiamloz•10mo ago

so it should be going to stderr, however, I dont know if this would be stder of the container

WillpowerOP•10mo ago

do i need modsecurity enabled for my proxy hosts by chance? im assuming not?

iiamloz•10mo ago

Not that shouldnt have an impact

WillpowerOP•10mo ago

found this within the container but its useless

WillpowerOP•10mo ago

its an error.log

WillpowerOP•10mo ago

iiamloz•10mo ago

yeah the container stdout will be just the proxy manager logs rather than nginx and this is /opt/npm/etc/crowdsec/crowdsec.conf?

WillpowerOP•10mo ago

for me its /data/etc/crowdsec in the container but yea

iiamloz•10mo ago

but you are persisting this directory on the host?

WillpowerOP•10mo ago

the file is the same when i cat it within the container console as on the host

WillpowerOP•10mo ago

iiamloz•10mo ago

Okay and since doing this did you destroy and recreate the container so it loads everything again?

WillpowerOP•10mo ago

like delete it and rerun my docker template (basically compose file)?

iiamloz•10mo ago

Yeah as typically you have to redeploy after changes as per npmplus instructions

WillpowerOP•10mo ago

just did that and no dice i dont see anything in the crowdsec logs about connections from 172.18.0.3 which is the npmplus container

WillpowerOP•10mo ago

its reading the logs tho?

WillpowerOP•10mo ago

I can dm you the crowdsec logs if needed

WillpowerOP•10mo ago

I did read a couple discussions about this on the git https://github.com/ZoeyVid/NPMplus/discussions/691 https://github.com/ZoeyVid/NPMplus/discussions/961#discussioncomment-10113769

GitHub

Why network_mode: host? · ZoeyVid NPMplus · Discussion #691

Can you elaborate on why you say "this fork has a dependency on network_mode: host" and similar in the readme? I'd like to avoid using network_mode: host for security and maintainabil...

GitHub

crowdsec config · ZoeyVid NPMplus · Discussion #961

Hello, i've setup npmplus and crowdsec according to the docs, ... i think im not sure its OK, changing mode from bridge to host doesnt seem to have an effect, so i'm not quite sure its effi...

WillpowerOP•10mo ago

i already did the ip changes and enabled ipv6 as they recommended before i came here ok, what the fuck i think i got it solved, or at least made some progress it seems if you use the unraid web ui text editor it encodes the npmplus crowdsec.conf incorrectly. after editing it in vim within the container i saw that it was adding ^M at the end of each line... deleting those and restarting the containers has it communicating with the api now i gotta see if its blocking properly it seems to work now api wise. I want to leave this open for a bit while i try to get extra parsers setup @iiamloz I got it working with the jellyfin collection and it successfully blocks ips. How can i test that it will block ips based on nginx docker integration/ logs? i tried doing some curl commands but it didnt pick up on it I just got 404 errors in the console is there any reason to have ipv6 enabled?

iiamloz•10mo ago

You can check cscli metrics and if it seems to be parsing and you see buckets stats then this is okay, if you see nothing then there probably an issue,.

WillpowerOP•10mo ago

@iiamloz

WillpowerOP•10mo ago

I am completely stuck getting it to parse nginx access.log. It works perfectly when using the explain command but fails to even show the log file in the Source sections of metrics

iiamloz•10mo ago

Did you mount the log files to the crowdsec container?

WillpowerOP•10mo ago

i did and im able to view them from within the crowdsec container

WillpowerOP•10mo ago

the strangest thing to me is when using the explain command it seems to detect the test traffic? cscli explain --file /var/log/npmplus/access.log --type npmplus

WillpowerOP•10mo ago

but when i run metrics

WillpowerOP•10mo ago

i was reading some posts on git where you were helping people with a similar issue but im not sure that advice applied to my situation

WillpowerOP•10mo ago

the strangest thing to me is it says its loaded

WillpowerOP•10mo ago

ive tried this with and without the header --- any other files that would be helpful to see? @iiamloz

iiamloz•10mo ago

Hmmm, if you cat the file does it have entries that are added since crowdsec was running?

WillpowerOP•10mo ago

the file updates live with a tail -f access.log from within the crowdsec container

iiamloz•10mo ago

Okay within your acquis.yaml file, where you defined filesnames can you add this key to yaml https://docs.crowdsec.net/docs/next/log_processor/data_sources/file#poll_without_inotify

filenames:
  - /var/log/npnplus/access.log
poll_without_inotify: true
....

filenames:
  - /var/log/npnplus/access.log
poll_without_inotify: true
....

as example

File(s) | CrowdSec

This module allows the Security Engine to acquire logs from text files (in one-shot and streaming mode), and GZ files in one-shot mode.

WillpowerOP•9mo ago

I think that might have fixed it!

WillpowerOP•9mo ago

Running this triggered an appsec ban so I’d assume that used nginx logs? for endpoint in admin login.php wp-login.php .env .git/config; do curl -s "https://subdomain.domain.com/$endpoint" > /dev/null & done wait @iiamloz

iiamloz•9mo ago

https://tenor.com/view/cat-cat-fun-fun-cat-cats-cattitude-gif-1504680328111330458

Tenor

iiamloz•9mo ago

this would of most likely triggered a normal ban, cause of .git/config and .env and then this would of triggered appsec scenario since it trigger 2 distinct appsec rules

WillpowerOP•9mo ago

What would be a good way to try and trigger a ban based off the nginx logs then? The line you told me to add definitely made a difference because before it wouldn’t show that file at all under source It is also parsing

iiamloz•9mo ago

for i in $(seq 1 50); do curl -s "https://subdomain.domain.com/$i" > /dev/null &;done

for i in $(seq 1 50); do curl -s "https://subdomain.domain.com/$i" > /dev/null &;done

WillpowerOP•9mo ago

Thanks

iiamloz•9mo ago

should trigger at least http probing however, make sure to run this on a machine you dont mind getting ban decisions

WillpowerOP•9mo ago

I’m doing it over vpn on another machine offsite. Not worried about that

WillpowerOP•9mo ago

The top is what triggered from that script

WillpowerOP•9mo ago

I’m assuming it’s all working now. Thanks a lot man!

iiamloz•9mo ago

Yep that worked!

WillpowerOP•9mo ago

It took a lot of figuring out but I’m glad I finally got it lol

iiamloz•9mo ago

well at least your name is true 💪 😄

WillpowerOP•9mo ago

@iiamloz How can I setup discord notifications for CAPI sourced denials/blocks? I have it working for bans based on sources (logs and dockers). I was watching the logs for npmplus (ngnix proxy manager) and saw an ip was blocked that i didnt get a notification for (first screenshot). after looking into it a bit, it was sourced from the CAPI and only shows up in decisions with the command above. running just cscli decisions list doesnt show it

Gaming

Programming

NPMplus won’t connect to api

Did you find this page helpful?