57 replies

Need help configuring Custom Provider (Chutes/OpenAI-compatible) for Vision & Audio - "Image Tool" c

I am self-hosting OpenClaw via Coolify (Docker). I am trying to use Chutes.ai as a custom provider for all models:

* Text:

openai/gpt-oss-120b-TEE

openai/gpt-oss-120b-TEE

(Working)
* Vision:

Qwen/Qwen3-VL-235B-A22B-Instruct

Qwen/Qwen3-VL-235B-A22B-Instruct

(Compatible via

openai-completions

openai-completions

API)
* Audio:

openai/whisper-large-v3

openai/whisper-large-v3

The Problem:
Text works perfectly. However, when I upload an image, two things go wrong:

1. Background Understanding Error: OpenClaw automatically tries to caption the image using

gpt-5-mini

gpt-5-mini

(default), which fails with

401 Incorrect API Key

401 Incorrect API Key

because it's sending my custom Chutes key to OpenAI.
* Question: How do I correctly point the

media.understanding

media.understanding

task to my custom Qwen model, or how do I fully disable it via config?

2. Tool Crash: The Agent doesn't seem to recognize Qwen as a native vision model. It tries to use the internal

image

image

tool instead, which crashes the container:

[tools] image failed: Failed to optimize image

[tools] image failed: Failed to optimize image

Error: Failed to optimize image at optimizeImageToJpeg...

Error: Failed to optimize image at optimizeImageToJpeg...

What I've Tried:
I attempted to configure the

models

models

block with separate providers for Vision and Audio, and I tried adding

imageModel

imageModel

to the

agents.defaults

agents.defaults

block, but it keeps falling back to the default OpenAI models or crashing the tool.

My Goal:
I need a working

openclaw.json

openclaw.json

snippet that properly registers a custom OpenAI-compatible Vision model (Qwen) so the Agent uses it natively (without invoking the "Image Tool" that crashes) and avoids the

gpt-5-mini

gpt-5-mini

fallback.

Current Config Attempt:

{
  "models": {
    "providers": {
      "chutes-main": {
        "baseUrl": "https://llm.chutes.ai/v1",
        "api": "openai-completions",
        "models": [
          { "id": "Qwen/Qwen3-VL-235B-A22B-Instruct", "name": "Qwen", "input": ["text", "image"] }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": { "primary": "chutes-main/..." },
      "imageModel": { "primary": "chutes-main/Qwen..." }
    }
  }
}

{
  "models": {
    "providers": {
      "chutes-main": {
        "baseUrl": "https://llm.chutes.ai/v1",
        "api": "openai-completions",
        "models": [
          { "id": "Qwen/Qwen3-VL-235B-A22B-Instruct", "name": "Qwen", "input": ["text", "image"] }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": { "primary": "chutes-main/..." },
      "imageModel": { "primary": "chutes-main/Qwen..." }
    }
  }
}

FotC�

Friends of the Crustacean 🦞🤝•3w ago•

57 replies

ventriquo

Need help configuring Custom Provider (Chutes/OpenAI-compatible) for Vision & Audio - "Image Tool" c

I am self-hosting OpenClaw via Coolify (Docker). I am trying to use Chutes.ai as a custom provider for all models:

* Text:

openai/gpt-oss-120b-TEE

openai/gpt-oss-120b-TEE

(Working)
* Vision:

Qwen/Qwen3-VL-235B-A22B-Instruct

Qwen/Qwen3-VL-235B-A22B-Instruct

(Compatible via

openai-completions

openai-completions

API)
* Audio:

openai/whisper-large-v3

openai/whisper-large-v3

The Problem:
Text works perfectly. However, when I upload an image, two things go wrong:

1. Background Understanding Error: OpenClaw automatically tries to caption the image using

gpt-5-mini

gpt-5-mini

(default), which fails with

401 Incorrect API Key

401 Incorrect API Key

because it's sending my custom Chutes key to OpenAI.
* Question: How do I correctly point the

media.understanding

media.understanding

task to my custom Qwen model, or how do I fully disable it via config?

2. Tool Crash: The Agent doesn't seem to recognize Qwen as a native vision model. It tries to use the internal

image

image

tool instead, which crashes the container:

[tools] image failed: Failed to optimize image

[tools] image failed: Failed to optimize image

Error: Failed to optimize image at optimizeImageToJpeg...

Error: Failed to optimize image at optimizeImageToJpeg...

What I've Tried:
I attempted to configure the

models

models

block with separate providers for Vision and Audio, and I tried adding

imageModel

imageModel

to the

agents.defaults

agents.defaults

block, but it keeps falling back to the default OpenAI models or crashing the tool.

My Goal:
I need a working

openclaw.json

openclaw.json

snippet that properly registers a custom OpenAI-compatible Vision model (Qwen) so the Agent uses it natively (without invoking the "Image Tool" that crashes) and avoids the

gpt-5-mini

gpt-5-mini

fallback.

Current Config Attempt:

{
  "models": {
    "providers": {
      "chutes-main": {
        "baseUrl": "https://llm.chutes.ai/v1",
        "api": "openai-completions",
        "models": [
          { "id": "Qwen/Qwen3-VL-235B-A22B-Instruct", "name": "Qwen", "input": ["text", "image"] }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": { "primary": "chutes-main/..." },
      "imageModel": { "primary": "chutes-main/Qwen..." }
    }
  }
}

{
  "models": {
    "providers": {
      "chutes-main": {
        "baseUrl": "https://llm.chutes.ai/v1",
        "api": "openai-completions",
        "models": [
          { "id": "Qwen/Qwen3-VL-235B-A22B-Instruct", "name": "Qwen", "input": ["text", "image"] }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": { "primary": "chutes-main/..." },
      "imageModel": { "primary": "chutes-main/Qwen..." }
    }
  }
}

Need help configuring Custom Provider (Chutes/OpenAI-compatible) for Vision & Audio - "Image Tool" c

Similar Threads

Need help configuring Custom Provider (Chutes/OpenAI-compatible) for Vision & Audio - "Image Tool" c

Similar Threads

Similar Threads

Similar Threads