Is this stupid? - Lie detector
Feature Request: Outbound Message Verification Hook
I’m a non technical founder and have build a pretty cool performance monitoring layer for my 35 person biz.
Problem is Skippy, open claw is a dirty liar.
I wonder…
Would it be possible to intercept every outbound agent message before it reaches me, run a verification script against it, and append a trust score? Something like a hook in the message pipeline that receives the draft, checks claims against the session's tool call log, and annotates or sends it back automatically requiring the LLM to actually do the thing it’s saying it’s done.
This doesn't exist in OpenClaw today and I can't build it externally.
Why:
An LLM has no internal mechanism to distinguish "I did this" from "I'm confident I did this."
Over weeks of production use, the fabrication compounds. Status reports contain phantom claims. Files reported as created don't exist. Scripts are "running" but were never executed.
So my agent and I built two deterministic verification scripts (zero LLM in the loop):
1. Tool Call Auditor — reads session JSONL, extracts every tool call, checks reality. Did the file get created? Did the edit land? Did the script error silently?
2. Claim Scanner — scans the agent's text for commitment language ("done", "running", "live"), checks for a corresponding tool call nearby. First run: 19% phantom rate. One in five claims had no tool call behind it.
These run on a 4-hour cron and catch problems after the fact. But I've already eaten the bullshit by then. The missing piece is real-time verification on every message before it reaches me.
It seems every AI lab is racing to make models smarter. Nobody is building the verification layer. This should be default infrastructure, not something a non-technical user has to duct-tape together.
I’m a non technical founder and have build a pretty cool performance monitoring layer for my 35 person biz.
Problem is Skippy, open claw is a dirty liar.
I wonder…
Would it be possible to intercept every outbound agent message before it reaches me, run a verification script against it, and append a trust score? Something like a hook in the message pipeline that receives the draft, checks claims against the session's tool call log, and annotates or sends it back automatically requiring the LLM to actually do the thing it’s saying it’s done.
This doesn't exist in OpenClaw today and I can't build it externally.
Why:
An LLM has no internal mechanism to distinguish "I did this" from "I'm confident I did this."
Over weeks of production use, the fabrication compounds. Status reports contain phantom claims. Files reported as created don't exist. Scripts are "running" but were never executed.
So my agent and I built two deterministic verification scripts (zero LLM in the loop):
1. Tool Call Auditor — reads session JSONL, extracts every tool call, checks reality. Did the file get created? Did the edit land? Did the script error silently?
2. Claim Scanner — scans the agent's text for commitment language ("done", "running", "live"), checks for a corresponding tool call nearby. First run: 19% phantom rate. One in five claims had no tool call behind it.
These run on a 4-hour cron and catch problems after the fact. But I've already eaten the bullshit by then. The missing piece is real-time verification on every message before it reaches me.
It seems every AI lab is racing to make models smarter. Nobody is building the verification layer. This should be default infrastructure, not something a non-technical user has to duct-tape together.
The AI that actually does things. Emails, calendar, home automation - all from your favorite chat app. New shell, ready to help. The lobster way.
134,723Members
Resources
Similar Threads
Was this page helpful?
