CrowdSec•6mo ago

Crowdsec + PG on K8S: agent can’t connect lapi

Hi, once configured to use a PG instance (schema is correctly created), the LAPI pod is in Running state, but all the agents are stuck with such errors in logs:

9 Replies

CrowdSec•6mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

ookOP•6mo ago

time="2025-03-28T08:35:49Z" level=info msg="max attempts reached for status code 401"
time="2025-03-28T08:35:49Z" level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-dfpgt): API error: ent: machine not found"

time="2025-03-28T08:35:49Z" level=info msg="max attempts reached for status code 401"
time="2025-03-28T08:35:49Z" level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-dfpgt): API error: ent: machine not found"

iiamloz•6mo ago

Could you provide your values.yaml? You can redact any information, I just need to see where you have configured the databases for both LAPI and agents.

ookOP•6mo ago

sure:

container_runtime: docker
agent:
  # Specify each pod whose logs you want to process
  acquisition:
  # The namespace where the pod is located
    - namespace: default
    # The pod name
      podName: traefik-ingress-controller-*
      # as in crowdsec configuration, we need to specify the program name to find a matching parser
      program: traefik
  additionalAcquisition:
    - source: file
      filenames:
      - /traefik-log/*.log
      labels:
        type: traefik
  env:
    - name: PARSERS
      value: "crowdsecurity/cri-logs"
    - name: COLLECTIONS
      value: "crowdsecurity/traefik"
config:
  config.yaml.local: |
    db_config:
      type:     postgresql
      user:     crowdsec
      password: <REDACTED>
      db_name:  crowdsec
      host:     10.3.99.5
      port:     5432
lapi:
  dashboard:
    enabled: false
  env:
    # To enroll the Security Engine to the console
    - name: ENROLL_KEY
      value: "<REDACTED>"
    - name: ENROLL_INSTANCE_NAME
      value: "neobok"
    - name: ENROLL_TAGS
      value: "k8s linux"
    - name: BOUNCER_KEY_TRAEFIK
      value: "<REDACTED>"
  persistentVolume:
    data:
      storageClassName: "direct-csi-min-io"
    config:
      storageClassName: "direct-csi-min-io"

container_runtime: docker
agent:
  # Specify each pod whose logs you want to process
  acquisition:
  # The namespace where the pod is located
    - namespace: default
    # The pod name
      podName: traefik-ingress-controller-*
      # as in crowdsec configuration, we need to specify the program name to find a matching parser
      program: traefik
  additionalAcquisition:
    - source: file
      filenames:
      - /traefik-log/*.log
      labels:
        type: traefik
  env:
    - name: PARSERS
      value: "crowdsecurity/cri-logs"
    - name: COLLECTIONS
      value: "crowdsecurity/traefik"
config:
  config.yaml.local: |
    db_config:
      type:     postgresql
      user:     crowdsec
      password: <REDACTED>
      db_name:  crowdsec
      host:     10.3.99.5
      port:     5432
lapi:
  dashboard:
    enabled: false
  env:
    # To enroll the Security Engine to the console
    - name: ENROLL_KEY
      value: "<REDACTED>"
    - name: ENROLL_INSTANCE_NAME
      value: "neobok"
    - name: ENROLL_TAGS
      value: "k8s linux"
    - name: BOUNCER_KEY_TRAEFIK
      value: "<REDACTED>"
  persistentVolume:
    data:
      storageClassName: "direct-csi-min-io"
    config:
      storageClassName: "direct-csi-min-io"

iiamloz•6mo ago

When you changed the values did you delete the persistentVolume? it might be the persisted data is old and it cant detect that anything has changed to generate new credentials

ookOP•6mo ago

hum, ok let’s shred all the setup so, installation shred, PV deleted, then fresh install. SE enrollement validated and… … same problem

time="2025-03-28T09:27:24Z" level=info msg="Loaded 52 scenarios"
time="2025-03-28T09:27:24Z" level=info msg="attempt 1 out of 2"
time="2025-03-28T09:27:24Z" level=info msg="attempt 2 out of 2"
time="2025-03-28T09:27:24Z" level=info msg="max attempts reached for status code 401"
time="2025-03-28T09:27:24Z" level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-s4656): API error: machine crowdsec-agent-s4656 not validated"

time="2025-03-28T09:27:24Z" level=info msg="Loaded 52 scenarios"
time="2025-03-28T09:27:24Z" level=info msg="attempt 1 out of 2"
time="2025-03-28T09:27:24Z" level=info msg="attempt 2 out of 2"
time="2025-03-28T09:27:24Z" level=info msg="max attempts reached for status code 401"
time="2025-03-28T09:27:24Z" level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (crowdsec-agent-s4656): API error: machine crowdsec-agent-s4656 not validated"

LAPI is running correctly but the agents can’t auth their watchers

blotus•6mo ago

It's a slightly different issue: by default, when registering a remote agent, it needs to be validated in LAPI. It's either a manual operation (not a great experience in k8s) or the agent can send a secret token to LAPI to automatically validate itself. It's done by default in the chart, but if you update config.yaml.local, you need to explicitely put the configuration:

  config.yaml.local: |
    api:
      server:
        auto_registration: # Activate if not using TLS for authentication
          enabled: true
          token: "${REGISTRATION_TOKEN}" # /!\ Do not modify this variable (auto-generated and handled by the chart)
          allowed_ranges: # /!\ Make sure to adapt to the pod IP ranges used by your cluster
            - "127.0.0.1/32"
            - "192.168.0.0/16"
            - "10.0.0.0/8"
            - "172.16.0.0/12"

  config.yaml.local: |
    api:
      server:
        auto_registration: # Activate if not using TLS for authentication
          enabled: true
          token: "${REGISTRATION_TOKEN}" # /!\ Do not modify this variable (auto-generated and handled by the chart)
          allowed_ranges: # /!\ Make sure to adapt to the pod IP ranges used by your cluster
            - "127.0.0.1/32"
            - "192.168.0.0/16"
            - "10.0.0.0/8"
            - "172.16.0.0/12"

(another solution is to use TLS certs for auth, but it's a bit more involved on your end)

ookOP•6mo ago

Indeed! Thanks, with this update and rotation of all agents, it’s running now. I’ll go on with recaptcha configuration /resolve

CrowdSec•6mo ago

Resolving Crowdsec + PG on K8S: agent can’t connect lapi This has now been resolved. If you think this is a mistake please run /unresolve

Gaming

Programming

Crowdsec + PG on K8S: agent can’t connect lapi

Did you find this page helpful?