vLLM Tool Calling not possible
I've spent the last three days getting tool calling to work with a local AI agent setup on an NVIDIA DGX Spark running Qwen3-Coder-Next-80B. The agent runs via OpenClaw and is accessible through Telegram.
The biggest hurdle was tool calling. OpenClaw has a known bug (#8923) with custom providers: the tools parameters don't get passed correctly to the model. No tools means no real agent behavior.
My workaround is a custom async SSE proxy written in Python with aiohttp, sitting between OpenClaw and vLLM. It intercepts requests, injects the missing tool definitions, and converts the response back into the expected streaming format. It works – but it adds latency compared to hitting vLLM directly, since every request has to go through an extra processing layer.
I'm currently waiting for bug #8923 to be fixed so I can drop the proxy entirely and go native. If anyone from the OpenClaw team reads this – please prioritize this one. It's the biggest blocker for anyone running local models with custom providers.
If you're in the same situation, the async proxy approach is a working workaround for now.
The biggest hurdle was tool calling. OpenClaw has a known bug (#8923) with custom providers: the tools parameters don't get passed correctly to the model. No tools means no real agent behavior.
My workaround is a custom async SSE proxy written in Python with aiohttp, sitting between OpenClaw and vLLM. It intercepts requests, injects the missing tool definitions, and converts the response back into the expected streaming format. It works – but it adds latency compared to hitting vLLM directly, since every request has to go through an extra processing layer.
I'm currently waiting for bug #8923 to be fixed so I can drop the proxy entirely and go native. If anyone from the OpenClaw team reads this – please prioritize this one. It's the biggest blocker for anyone running local models with custom providers.
If you're in the same situation, the async proxy approach is a working workaround for now.
The AI that actually does things. Emails, calendar, home automation - all from your favorite chat app. New shell, ready to help. The lobster way.
131,281Members
Resources
Similar Threads
Was this page helpful?
