CrowdSec•4d ago

Not receiving notifications from (not LAPI) machine

As the titel says... I have not been receiving notifications as of recently when an alert originated from my host 'tower'... Distributed setup, mix of containers and native installs, LAPI is a container. Instance not sending notifications is also a container... I don't see any relevant erros in the docker logs. However I have a feeling some parsing is failing and thus not sending anything? I have 2 notifications formats, discord & pushover:

format: |
  {
    "content": null,
    "embeds": [
      {{range . -}}
      {{$alert := . -}}
      {{range .Decisions -}}
      {{if $alert.Source.Cn -}}
      {
        "title": "{{$alert.MachineID}}: {{.Scenario}}",
        "description": ":flag_{{ $alert.Source.Cn | lower }}: {{$alert.Source.IP}} will get a {{.Type}} for the next {{.Duration}}.",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": "16711680"
      }
      {{end}}
      {{if not $alert.Source.Cn -}}
      {
        "title": "{{$alert.MachineID}}: {{.Scenario}}",
        "description": ":pirate_flag: {{$alert.Source.IP}} will get a {{.Type}} for the next {{.Duration}}.",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": "16711680"
      }
      {{end}}
      {{end -}}
      {{end -}}
    ]
  }
url: https://discord.com/api/webhooks/xxx
method: POST
headers:
  Content-Type: application/json

format: |
  {
    "content": null,
    "embeds": [
      {{range . -}}
      {{$alert := . -}}
      {{range .Decisions -}}
      {{if $alert.Source.Cn -}}
      {
        "title": "{{$alert.MachineID}}: {{.Scenario}}",
        "description": ":flag_{{ $alert.Source.Cn | lower }}: {{$alert.Source.IP}} will get a {{.Type}} for the next {{.Duration}}.",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": "16711680"
      }
      {{end}}
      {{if not $alert.Source.Cn -}}
      {
        "title": "{{$alert.MachineID}}: {{.Scenario}}",
        "description": ":pirate_flag: {{$alert.Source.IP}} will get a {{.Type}} for the next {{.Duration}}.",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": "16711680"
      }
      {{end}}
      {{end -}}
      {{end -}}
    ]
  }
url: https://discord.com/api/webhooks/xxx
method: POST
headers:
  Content-Type: application/json

format: |
  {
    "token": "xxx",
    "user": "xxx",
    "message": "{{range . -}}{{$alert := . -}}{{range .Decisions -}}{{if $alert.Source.Cn}}{{ $alert.Source.Cn | lower }} {{end}}<b><a href=\"https://www.shodan.io/search?query={{.Value}}\">{{.Value}}</a></b> will get a <b>{{.Type}}</b> for the next <b>{{.Duration}}</b>.<br/>{{end -}}{{end -}}",
    "html": "1",
    "title": "{{if gt (len .) 0}}{{ (index . 0).MachineID }}: {{ (index . 0).Scenario }}{{else}}Crowdsec{{end}}",
    "priority": "-2"
  }
url: https://api.pushover.net/1/messages.json

method: POST
headers:
  Content-Type: "application/json"

format: |
  {
    "token": "xxx",
    "user": "xxx",
    "message": "{{range . -}}{{$alert := . -}}{{range .Decisions -}}{{if $alert.Source.Cn}}{{ $alert.Source.Cn | lower }} {{end}}<b><a href=\"https://www.shodan.io/search?query={{.Value}}\">{{.Value}}</a></b> will get a <b>{{.Type}}</b> for the next <b>{{.Duration}}</b>.<br/>{{end -}}{{end -}}",
    "html": "1",
    "title": "{{if gt (len .) 0}}{{ (index . 0).MachineID }}: {{ (index . 0).Scenario }}{{else}}Crowdsec{{end}}",
    "priority": "-2"
  }
url: https://api.pushover.net/1/messages.json

method: POST
headers:
  Content-Type: "application/json"

10 Replies

CrowdSec•4d ago

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

blotus•4d ago

Hello, Notifications are sent by LAPI: they are handled by the profiles, which are only evaluated by LAPI when it receives an alert from a log processor

PintjesBierOP•4d ago

Yeah, I only have the notification profiles created on the LAPI.

blotus•4d ago

Can you share your profiles config ? Does the other machine appear in cscli machines list ? (run this on LAPI) Do you see any alerts from the machine in cscli alerts list ? (you can use the -m flag to show the name of the machine that generated the alert)

PintjesBierOP•3d ago

name: default_ip_remediation
#debug: true
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
  - type: ban
    duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
  - discord
  - pushover
on_success: break

name: default_ip_remediation
#debug: true
filters:
  - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
  - type: ban
    duration: 4h
#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
notifications:
  - discord
  - pushover
on_success: break

It does show up in the alerts:

root@Tower:~# docker exec crowdsec cscli alerts inspect 13440

################################################################################################

 - ID           : 13440
 - Date         : 2025-09-29T23:12:25Z
 - Machine      : tower
 - Simulation   : false
 - Remediation  : true
 - Reason       : crowdsecurity/iptables-scan-multi_ports

root@Tower:~# docker exec crowdsec cscli alerts inspect 13440

################################################################################################

 - ID           : 13440
 - Date         : 2025-09-29T23:12:25Z
 - Machine      : tower
 - Simulation   : false
 - Remediation  : true
 - Reason       : crowdsecurity/iptables-scan-multi_ports

PintjesBierOP•3d ago

&On the LAPI:

blotus•3d ago

strange, your configuration looks ok can you try cscli notifications reinject <alert_id> and see if it sent properly ?

PintjesBierOP•3d ago

╭─🎯 ubuntu on docker in ~ via 🐍 pyenv
╰─✗  sudo docker exec -it crowdsec cscli notifications reinject 13440
DEBU[0000] starting plugin                               args="[/usr/local/lib/crowdsec/plugins/notification-http]" path=/usr/local/lib/crowdsec/plugins/notification-http
DEBU[0000] plugin started                                path=/usr/local/lib/crowdsec/plugins/notification-http pid=3199
DEBU[0000] waiting for RPC address                       plugin=/usr/local/lib/crowdsec/plugins/notification-http
DEBU[0000] using plugin                                  version=1
TRAC[0000] waiting for stdio data
INFO registered plugin http_default
INFO registered plugin pushover
INFO registered plugin discord
INFO The profile default_ip_remediation matched, sending to its configured notification plugins
INFO The profile default_ip_remediation contains a 'on_success: break' so bailing out
INFO pluginTomb dying
INFO[0000] received signal for discord config            @module=http-plugin
INFO[0000] received signal for pushover config           @module=http-plugin
INFO killing all plugins
DEBU[0001] received EOF, stopping recv loop              err="rpc error: code = Unavailable desc = error reading from server: EOF"
INFO[0001] plugin process exited                         id=3199 plugin=/usr/local/lib/crowdsec/plugins/notification-http
DEBU[0001] plugin exited

╭─🎯 ubuntu on docker in ~ via 🐍 pyenv
╰─✗  sudo docker exec -it crowdsec cscli notifications reinject 13440
DEBU[0000] starting plugin                               args="[/usr/local/lib/crowdsec/plugins/notification-http]" path=/usr/local/lib/crowdsec/plugins/notification-http
DEBU[0000] plugin started                                path=/usr/local/lib/crowdsec/plugins/notification-http pid=3199
DEBU[0000] waiting for RPC address                       plugin=/usr/local/lib/crowdsec/plugins/notification-http
DEBU[0000] using plugin                                  version=1
TRAC[0000] waiting for stdio data
INFO registered plugin http_default
INFO registered plugin pushover
INFO registered plugin discord
INFO The profile default_ip_remediation matched, sending to its configured notification plugins
INFO The profile default_ip_remediation contains a 'on_success: break' so bailing out
INFO pluginTomb dying
INFO[0000] received signal for discord config            @module=http-plugin
INFO[0000] received signal for pushover config           @module=http-plugin
INFO killing all plugins
DEBU[0001] received EOF, stopping recv loop              err="rpc error: code = Unavailable desc = error reading from server: EOF"
INFO[0001] plugin process exited                         id=3199 plugin=/usr/local/lib/crowdsec/plugins/notification-http
DEBU[0001] plugin exited

That did actually send the message btw

blotus•3d ago

so it's working do you see any errors in LAPI logs at around the time of the last alert for this machine ? I'm thinking network error or something like that, cscli notification reinject is using the same code as LAPI does at runtime to send notifications

PintjesBierOP•3d ago

No errors afaik... Only those mentioned in #Failed to bind to json but I don't think these are related?

Gaming

Programming

Not receiving notifications from (not LAPI) machine

Did you find this page helpful?