ngx.timer error when loading decisions
Yesterday we've updated the Nginx ingress controller and Crowdsec on AKS.
Nginx ingress 12.1 by mmetc: https://github.com/crowdsecurity/cs-openresty-bouncer/issues/60
We had already tested this with a free account without issues on a low traffic staging site.
On the site we have a Crowdsec enterprise account.
New alerts and decisions are shown in the online console.
But there are error messages in NGINX and the LAPI (crowdsecurity/crowdsec:v1.6.10).
NGINX ingress errors:
2025/08/19 08:37:06 [error] 33#33: *730358 lua tcp socket read timed out, context: ngx.timer
2025/08/19 08:37:06 [error] 33#33: *730358 [lua] stream.lua:157: stream_query(): request to crowdsec lapi http://10.0.236.219:8080/v1/decisions/stream?startup=true failed: timeout, context: ngx.timer
2025/08/19 08:37:06 [error] 33#33: *730358 [lua] crowdsec.lua:342: Failed to query the stream: request to crowdsec lapi http://10.0.236.219:8080/v1/decisions/stream?startup=true failed: timeout, context: ngx.timer
LAPI errors:
time="2025-08-19T09:34:26Z" level=warning msg="QueryExpiredDecisionsWithFilters : context canceled"
time="2025-08-19T09:34:26Z" level=error msg="unable to query expired decision for 'nginx-controller@10.244.1.180' : expired decisions: unable to query"
There is no last API pull in the "cscli bouncers list".
A new bouncer was created with a new API key which was then added in the crowdsec-ingress-bouncer.yaml and redeployed.
In the online console there is the notification which we always get after updating crowdsec:
The organization has reached its full capacity for reserved slots.
This post seems related: https://discord.com/channels/921520481163673640/1342063343497773098/1342063343497773098
Best regards,
Gunnar
GitHub
Compatibility with nginx-ingress >= 1.12.0 · Issue #60 · crowdsec...
The ingress-nginx project has removed support for plugins (PR here). This prevents the remediation component from working, as the code is not even loaded anymore. We are exploring options to workar...
3 Replies
Important Information
This post has been marked as resolved. If this is a mistake please press the red button below or type
/unresolve© Created By WhyAydan for CrowdSec ❤️
What we've tried to solve it:
We've increased the timeout to 10 seconds in the crowdsec-ingress-bouncer.yaml:
- name: REQUEST_TIMEOUT
value: "10000"
And removed all decisions with "cscli decisions delete --all" since requesting "decisions list -a" manually made the LAPI pod run out of memory and restart.
But still there are the errors mentioned above.
There is a feature flag to chunk decisions but not sure how to enable this on Kubernetes (or if this could help):
chunked_decisions_stream
crowdsec-ingress-bouncer.yaml:
controller:
image:
registry: ghcr.io
image: mmetc/ingress-nginx/controller
tag: "v1.12.1"
digest: "sha256:b8cbb0d1d73e7516fe0d83d2228a16e2ada1f6a04fad61e1e345b52c7dda2705"
digestChroot: "sha256:8c051eb30ddd22bc9e610267b5d73af6558fb842af4124d86e0b48e181c0b460"
extraVolumes:
- name: crowdsec-bouncer-plugin
emptyDir: {}
extraInitContainers:
- name: init-clone-crowdsec-bouncer
image: crowdsecurity/lua-bouncer-plugin
imagePullPolicy: IfNotPresent
env:
- name: API_URL
value: "http://<service-ip>:8080"
- name: API_KEY
value: "***"
- name: BOUNCER_CONFIG
value: "/crowdsec/crowdsec-bouncer.conf"
- name: CAPTCHA_PROVIDER
value: "recaptcha" # valid providers are recaptcha, hcaptcha, turnstile
- name: BAN_TEMPLATE_PATH
value: /etc/nginx/lua/plugins/crowdsec/templates/ban.html
- name: CAPTCHA_TEMPLATE_PATH
value: /etc/nginx/lua/plugins/crowdsec/templates/captcha.html
- name: CACHE_EXPIRATION
value: "10"
- name: REQUEST_TIMEOUT
value: "10000"
- name: MODE
value: "stream"
- name: UPDATE_FREQUENCY
value: "10"
command: ["sh", "-c", "apk update; apk add bash; bash /docker_start.sh; mkdir -p /lua_plugins/crowdsec/; cp -R /crowdsec/* /lua_plugins/crowdsec/",]
volumeMounts:
- name: crowdsec-bouncer-plugin
mountPath: /lua_plugins
extraVolumeMounts:
- name: crowdsec-bouncer-plugin
mountPath: /etc/nginx/lua/plugins/crowdsec
subPath: crowdsec
config:
plugins: "crowdsec"
lua-shared-dicts: "crowdsec_cache: 50m"
crowdsec-values.yaml:
# Here you can specify your own custom configuration to be loaded in crowdsec agent or lapi
# Each config needs to be a multi-line using '|' in YAML specs
# for the agent those configs will be loaded : parsers, scenarios, postoverflows, simulation.yaml
# for the lapi those configs will be loaded : profiles.yaml, notifications, console.yaml
config:
# -- To better understand stages in parsers, you can take a look at https://docs.crowdsec.net/docs/next/parsers/intro/
parsers:
s02-enrich:
dkwhitelist.yaml: |
name: dk/whitelistdkoffice
description: "Whitelist ip"
whitelist:
reason: "whitelist events from dk"
ip:
- "<ip>"
cidr:
- "<cidr>"
# for raw logs format: json or cri (docker|containerd)
container_runtime: containerd
agent:
# Specify each pod whose logs you want to process
acquisition:
# The namespace where the pod is located
- namespace: ingress-nginx
# The pod name
podName: ingress-nginx-controller-*
# as in crowdsec configuration, we need to specify the program name to find a matching parser
program: nginx
poll_without_inotify: true
env:
- name: COLLECTIONS
value: "crowdsecurity/nginx crowdsecurity/wordpress"
- name: PARSERS
value: "crowdsecurity/nginx-logs"
- name: SCENARIOS
value: "crowdsecurity/http-bf-wordpress_bf_xmlrpc"
lapi:
env:
# To enroll the Security Engine to the console
- name: ENROLL_KEY
value: "<enroll_key>"
- name: ENROLL_INSTANCE_NAME
value: "<cluster_name>"
- name: ENROLL_TAGS
value: "k8s linux"
This issue is solved now.
I've added this in the crowdsec-values.yaml LAPI env:
lapi:
env:
- name: CROWDSEC_FEATURE_CHUNKED_DECISIONS_STREAM
value: 'true'Resolving ngx.timer error when loading decisions
This has now been resolved. If you think this is a mistake please run
/unresolve