Currently stream APIs (including official API and OpenAI compatible API) do not properly handle tool

B

Brandon StudioOP•3/31/25, 8:19 AM

Also, OpenAI compatible API only returns

null

null

in non-stream mode

V

vivsam•3/31/25, 8:38 AM

It'd be nice if you guys add Zero-Shot-Text-Classification models like https://huggingface.co/facebook/bart-large-mnli

facebook/bart-large-mnli · Hugging Face

S

sku•3/31/25, 5:46 PM

I've tried to make requests to several different models today over the REST API and I've only gotten this response. Are there any known issues?

{
    "result": null,
    "success": false,
    "errors": [
        {
            "code": 7010,
            "message": "Service unavailable"
        }
    ],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [
        {
            "code": 7010,
            "message": "Service unavailable"
        }
    ],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [
        {
            "code": 7010,
            "message": "Service unavailable"
        }
    ],
    "messages": []
}

{
    "result": null,
    "success": false,
    "errors": [
        {
            "code": 7010,
            "message": "Service unavailable"
        }
    ],
    "messages": []
}

BBrandon Studio Currently stream APIs (including official API and OpenAI compatible API) do not ...

I

Isaac McFadyen•3/31/25, 8:28 PM

Is this for AI Gateway or Workers AI?

I

Isaac McFadyen•3/31/25, 8:28 PM

And what models does this occur on?

N

nclevenger•3/31/25, 9:49 PM

hey - this new model is amazing ... it's SOTA on SWE Bench for a 32b model, scoring 37% which is SHOCKING for a model this small ... would love to see this on Workers AI https://huggingface.co/all-hands/openhands-lm-32b-v0.1

all-hands/openhands-lm-32b-v0.1 · Hugging Face

IIsaac McFadyen Is this for AI Gateway or Workers AI?

B

Brandon StudioOP•4/1/25, 2:09 AM

Workers AI, behaviors with and without AI Gateway are same

IIsaac McFadyen And what models does this occur on?

B

Brandon StudioOP•4/1/25, 2:12 AM

I see the two models (

@hf/nousresearch/hermes-2-pro-mistral-7b

@hf/nousresearch/hermes-2-pro-mistral-7b

@hf/nousresearch/hermes-2-pro-mistral-7b

@hf/nousresearch/hermes-2-pro-mistral-7b and

@cf/meta/llama-3.3-70b-instruct-fp8-fast

@cf/meta/llama-3.3-70b-instruct-fp8-fast

@cf/meta/llama-3.3-70b-instruct-fp8-fast

@cf/meta/llama-3.3-70b-instruct-fp8-fast) behaves differently.
Response of hermes does not handle

<tool_call>

<tool_call>

<tool_call>

<tool_call> label in stream mode, as shown in Image 1.
Response of llama 3.3 does not distinguish text content and tool call, as shown in Image 2.

B

Brandon StudioOP•4/1/25, 2:14 AM

I expect the tool call response to present in

tool_calls

tool_calls

tool_calls

tool_calls field rather than

response

response

field, just like non-stream mode

Ssku I've tried to make requests to several different models today over the REST API ...

G

glncy•4/1/25, 4:52 AM

same i am gettings this error on my local. but when i try to deploy a sample workers ai, it's working.

Original message was deleted

I

Isaac McFadyen•4/1/25, 3:44 PM

That is for Cloudflare Stream which is a different product.

I

Isaac McFadyen•4/1/25, 3:44 PM

I know for sure it accepts WAV 16khz mono, but I'm not sure about other formats.

G

glncy•4/1/25, 6:10 PM

how do i access workers ai locally? i have this nextjs app. using bindings and it returns me service unavailable 7010. even on rest api, it returns me the same response. i've been stuck for two days about this :3

Ssku I've tried to make requests to several different models today over the REST API ...

G

glncy•4/1/25, 7:06 PM

hi @sku want to know if you already figured this out? thanks

Mmichelle this seems to be a bug from an upstream service, we're taking a look

G

glncy•4/1/25, 7:34 PM

hi! thank you for your help. i already sent my account ID to you.

M

mr.niko.la•4/2/25, 8:16 PM

The whisper v3 turbo has 25mb file limit ? Would be nice if the website has this details ……

S

Spencer Hurst•4/3/25, 9:10 PM

Hello. Does anybody have experience using a custom lora on Workers AI? I uploaded the lora files and the finetune is listed via "wrangler ai finetune list", but when I try and do inference via curl with the finetune ID reported by wrangler, it says finetune not found. The adapter works on huggingface with the same files, and I can do inference on the base model (mistral 7b instruct v0.2) via curl fine. Does anybody know the right way to do this?

This is the error I'm getting (all redacted fields filled in properly):

$ curl https://api.cloudflare.com/client/v4/accounts/MY_ACCOUNT_ID/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora \
  -H 'Authorization: Bearer MY_API_TOKEN' \
  -d '{
    "messages": [{"role": "user", "content": "Hello world"}],
    "raw": true,
    "lora": "MY_LORA_ID"
  }'
{"errors":[{"message":"AiError: AiError: Finetune MY_LORA_ID not found (07af68df-18b6-4a5d-bfeb-3453830da3be)","code":3037}],"success":false,"result":{},"
messages":[]}

$ curl https://api.cloudflare.com/client/v4/accounts/MY_ACCOUNT_ID/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora \
  -H 'Authorization: Bearer MY_API_TOKEN' \
  -d '{
    "messages": [{"role": "user", "content": "Hello world"}],
    "raw": true,
    "lora": "MY_LORA_ID"
  }'
{"errors":[{"message":"AiError: AiError: Finetune MY_LORA_ID not found (07af68df-18b6-4a5d-bfeb-3453830da3be)","code":3037}],"success":false,"result":{},"
messages":[]}

$ curl https://api.cloudflare.com/client/v4/accounts/MY_ACCOUNT_ID/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora \
  -H 'Authorization: Bearer MY_API_TOKEN' \
  -d '{
    "messages": [{"role": "user", "content": "Hello world"}],
    "raw": true,
    "lora": "MY_LORA_ID"
  }'
{"errors":[{"message":"AiError: AiError: Finetune MY_LORA_ID not found (07af68df-18b6-4a5d-bfeb-3453830da3be)","code":3037}],"success":false,"result":{},"
messages":[]}

$ curl https://api.cloudflare.com/client/v4/accounts/MY_ACCOUNT_ID/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora \
  -H 'Authorization: Bearer MY_API_TOKEN' \
  -d '{
    "messages": [{"role": "user", "content": "Hello world"}],
    "raw": true,
    "lora": "MY_LORA_ID"
  }'
{"errors":[{"message":"AiError: AiError: Finetune MY_LORA_ID not found (07af68df-18b6-4a5d-bfeb-3453830da3be)","code":3037}],"success":false,"result":{},"
messages":[]}

S

s•4/4/25, 4:57 AM

S

sillah•4/4/25, 10:30 AM

Hello, I have recently deployed cloudfare worker. All seems to work well however I can't seem to set max tokens, it always crops output to 256 tokens. Anyone who has resolved this issue curl -X POST https://xxx \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "explain c++ in detail?"}
],
"max_tokens": 500
}'
Response :
{"inputs":{"messages":[{"role":"user","content":"explain c++ in detail?"}]},"response":{"response":"<think>\nOkay, so I'm trying to understand C++ in detail. I've heard it's a powerful programming language, but I'm not exactly sure how it all fits together. Let me start by breaking down what I know and what I need to figure out.\n\nFirst, I know that C++ is an extension of C. So, it includes everything from C but adds more features. But what exactly are those features? I remember something about classes and objects, which makes me think it's object-oriented. That must mean it's good for building large applications where you can structure code more cleanly.\n\nThe assistant mentioned that C++ is compiled, which I think means it's faster because it's converted directly into machine code. But I'm not entirely sure how that process works compared to interpreted languages. Maybe I should look into the compilation steps, like preprocessing, compiling, assembling, and linking. Understanding that could help me see why C++ is efficient.\n\nI'm also a bit confused about the Standard Template Library (STL). I know it's part of C++, but how does it help? It sounds like it provides a lot of useful data structures and algorithms, which would save time when coding. But I'm not sure how to use it effectively or what the main","usage":{"prompt_tokens":13,"completion_tokens":256,"total_tokens":269}}}

N

nclevenger•4/6/25, 6:43 AM

Will Llama 4 Scout on Workers AI be offered via OpenRouter?

https://openrouter.ai/meta-llama/llama-4-scout
https://developers.cloudflare.com/workers-ai/models/llama-4-scout-17b-16e-instruct/

Llama 4 Scout - API, Providers, Stats

Llama 4 Scout 17B Instruct (16E) is a mixture-of-experts (MoE) language model developed by Meta, activating 17 billion parameters out of a total of 109B. It supports native multimodal input (text and image) and multilingual output (text and code) across 12 supported languages. Run Llama 4 Scout with API

Cloudflare Docs

llama-4-scout-17b-16e-instruct · Cloudflare Workers AI docs

Meta's Llama 4 Scout is a 17 billion parameter model with 16 experts that is natively multimodal. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.

Nnclevenger Will Llama 4 Scout on Workers AI be offered via OpenRouter? https://openrouter....

S

Silvan•4/6/25, 2:29 PM

i wondered the same but workers ai is currently not working (anymore) with openrouter

S

Silvan•4/6/25, 2:30 PM

im assuming it will be when it returns

J

Jason Barry•4/7/25, 1:39 AM

humble request to redesign https://developers.cloudflare.com/workers-ai/models/ into a simple table that shows model name, context window size, price per input/output token, support for JSON mode, etc. I'm finding I need to drill into each detail page to compare models. I'd prefer to compare at a birds-eye level.

JJason Barry humble request to redesign https://developers.cloudflare.com/workers-ai/models/ ...

M

mr.niko.la•4/7/25, 7:17 AM

Ya very confusing.

I can’t even find whisper v3 turbo sample code on how to test it or even what the file size limit is or if I will need to chunk it….

N

namlongday•4/7/25, 10:51 AM

hello claufalre plan business f

L

LrGian•4/7/25, 4:46 PM

Hello, I want to do a proof of concept (PoC) on an AI worker. How much budget do you think I can make for at least 2 models? I'm looking to find out if it meets the data I need for my application.an approximate

R

r4v4rg4s•4/7/25, 6:46 PM

Are there any plans for AutoRag service to support multimodal embeddings?

Mmr.niko.la Ya very confusing. I can’t even find whisper v3 turbo sample code on how to tes...

M

mr.niko.la•4/7/25, 8:12 PM

WHAT IS THE WHISPER V3 TURBO FILE SIZE LIMIT ON WORKERS AI!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

面

面条•4/8/25, 2:22 AM

Are there plans to support some Qwen LLM models?

J

Juan•4/8/25, 7:07 PM

Hello, what's the best channel to ask questions about the Agents API?

A

a4amaan•4/8/25, 10:28 PM

Hello, i have mixed text

AMAAN KHAN Mobile: (+92) 321 322 578 Email: a4amaan@asd.com SUMMARY: ▪ 7+ Years of Software Development Experience. ▪ Experience in design, development of web based applications. ▪ Experience in Python: Django, Flask, Django Rest Framework, JWT Authentication, SaaS Based Applications. ▪ Experience in Web Development: HTML, CSS, Javascript, Nodejs, Express, Mysql, Mongodb ▪ Experience Developing SPA in VUE Js, VUEX, LocalStorage along with Rest APIs in Django Rest Framework. ▪ Experience in

AMAAN KHAN Mobile: (+92) 321 322 578 Email: a4amaan@asd.com SUMMARY: ▪ 7+ Years of Software Development Experience. ▪ Experience in design, development of web based applications. ▪ Experience in Python: Django, Flask, Django Rest Framework, JWT Authentication, SaaS Based Applications. ▪ Experience in Web Development: HTML, CSS, Javascript, Nodejs, Express, Mysql, Mongodb ▪ Experience Developing SPA in VUE Js, VUEX, LocalStorage along with Rest APIs in Django Rest Framework. ▪ Experience in

AMAAN KHAN Mobile: (+92) 321 322 578 Email: a4amaan@asd.com SUMMARY: ▪ 7+ Years of Software Development Experience. ▪ Experience in design, development of web based applications. ▪ Experience in Python: Django, Flask, Django Rest Framework, JWT Authentication, SaaS Based Applications. ▪ Experience in Web Development: HTML, CSS, Javascript, Nodejs, Express, Mysql, Mongodb ▪ Experience Developing SPA in VUE Js, VUEX, LocalStorage along with Rest APIs in Django Rest Framework. ▪ Experience in

AMAAN KHAN Mobile: (+92) 321 322 578 Email: a4amaan@asd.com SUMMARY: ▪ 7+ Years of Software Development Experience. ▪ Experience in design, development of web based applications. ▪ Experience in Python: Django, Flask, Django Rest Framework, JWT Authentication, SaaS Based Applications. ▪ Experience in Web Development: HTML, CSS, Javascript, Nodejs, Express, Mysql, Mongodb ▪ Experience Developing SPA in VUE Js, VUEX, LocalStorage along with Rest APIs in Django Rest Framework. ▪ Experience in

is there anyway to get specific data from this text like name email and contact using any ai model? i mean in proper json format

A

andyt•4/9/25, 8:47 AM

Hey is there a file size limit for toMarkdown? I have tried a few larger files that are failing

Aa4amaan Hello, i have mixed text ```AMAAN KHAN Mobile: (+92) 321 322 578 Email: a4amaan@...

A

andyt•4/9/25, 8:47 AM

i think this is what your looking for, https://developers.cloudflare.com/workers-ai/features/json-mode/

Cloudflare Docs

JSON Mode · Cloudflare Workers AI docs

When we want text-generation AI models to interact with databases, services, and external systems programmatically, typically when using tool calling or building AI agents, we must have structured response formats rather than natural language.

M

mongj•4/9/25, 9:57 AM

hi! what's the difference between @cf/openai/whisper-large-v3-turbo and @cf/openai/whisper? which whisper model is the latter using?

Aandyt i think this is what your looking for, https://developers.cloudflare.com/workers...

A

a4amaan•4/9/25, 11:11 AM

Really helpfull thanks...

M

Muhammadreza Haghiri•4/9/25, 12:12 PM

it seems flux schnell is down.

M

Muhammadreza Haghiri•4/9/25, 12:24 PM

It takes so long and nothing happens.

Aandyt Hey is there a file size limit for toMarkdown? I have tried a few larger files t...

A

andyt•4/9/25, 1:49 PM

found this, assuming these limist for AutoRag https://developers.cloudflare.com/autorag/configuration/data-source/

Cloudflare Docs

Data source · AutoRAG

AutoRAG currently supports Cloudflare R2 as the data source for storing your knowledge base. To get started, configure an R2 bucket containing your data.

M

mongj•4/9/25, 7:23 PM

will the whisper model on workers ai eventually support auto language detection? we would like to move from openai to workers ai but the lack of language detection is the biggest blocker right now

M

mongj•4/9/25, 7:30 PM

it's certainly possible to put a language detection model in front, but that defeats the cost and latency benefit from using the serverless models on workers ai, especially since other whisper providers (openai, fireworks, groq) already provides it

S

sjg•4/9/25, 9:50 PM

Has anyone used https://developers.cloudflare.com/workers-ai/markdown-conversion/ via REST API? I've not been able to create a token that works.

Cloudflare Docs

Markdown Conversion · Cloudflare Workers AI docs

Markdown is essential for text generation and large language models (LLMs) in training and inference because it can provide structured, semantic, human, and machine-readable input. Likewise, Markdown facilitates chunking and structuring input data for better retrieval and synthesis in the context of RAGs, and its simplicity and ease of parsing a...

R

Rapid•4/9/25, 10:09 PM

Hiya, I'm trying to understand waht should I use, my case is as follow:

I have an API that receives X amount of parameters, I want the user to be able to write "buy me X with Y value and then do Z actions", I'm basically looking for a way for an AI to interept that and return a json like {param1: x, param2: y, param3:z}.

What should I use for that?

R

Rapid•4/9/25, 10:13 PM

Is JSON mode the best solution for that?

Ssjg Has anyone used https://developers.cloudflare.com/workers-ai/markdown-conversion...

S

sjg•4/9/25, 10:15 PM

I created an API token with AI Workers and AutoRAG permissions, no longer get

{"code":10000,"message":"Authentication error"}

{"code":10000,"message":"Authentication error"}

in the REST response, but do get

{"message":"Server Error","code":6001}

{"message":"Server Error","code":6001}

, anyone seen this error before?

C

can•4/9/25, 10:53 PM

hello y'all, is there anybody who is using AI SDK with Cloudflare Workers AI provider or anybody who is working on the provider atm?

I was surprised when I saw that image-part is not implemented on the provider, so I was wondering if I should implement it and make a PR or is there anything to be discussed first?

A

a4amaan•4/9/25, 11:52 PM

Trying to return stream from worker

const answer = await ctx.env.AI.autorag("ats-rag").aiSearch({
    query: 'Who is Amaan Khan',
    // model: "@cf/meta/llama-3-8b-instruct",
    model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
    rewrite_query: true,
    max_num_results: 10,
    ranking_options: {
        score_threshold: 0.4,
    },
    stream: true
});
return new Response(answer, {
    headers: {
        'Content-Type': 'text/event-stream'
    }
});

const answer = await ctx.env.AI.autorag("ats-rag").aiSearch({
    query: 'Who is Amaan Khan',
    // model: "@cf/meta/llama-3-8b-instruct",
    model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
    rewrite_query: true,
    max_num_results: 10,
    ranking_options: {
        score_threshold: 0.4,
    },
    stream: true
});
return new Response(answer, {
    headers: {
        'Content-Type': 'text/event-stream'
    }
});

const answer = await ctx.env.AI.autorag("ats-rag").aiSearch({
    query: 'Who is Amaan Khan',
    // model: "@cf/meta/llama-3-8b-instruct",
    model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
    rewrite_query: true,
    max_num_results: 10,
    ranking_options: {
        score_threshold: 0.4,
    },
    stream: true
});
return new Response(answer, {
    headers: {
        'Content-Type': 'text/event-stream'
    }
});

const answer = await ctx.env.AI.autorag("ats-rag").aiSearch({
    query: 'Who is Amaan Khan',
    // model: "@cf/meta/llama-3-8b-instruct",
    model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
    rewrite_query: true,
    max_num_results: 10,
    ranking_options: {
        score_threshold: 0.4,
    },
    stream: true
});
return new Response(answer, {
    headers: {
        'Content-Type': 'text/event-stream'
    }
});

getting error

"logs": [
    {
      "message": [
        "SyntaxError: Unexpected token 'd', \"data: {\"no\"... is not valid JSON"
      ],
      "level": "error",
      "timestamp": 1744242318653
    }
  ]

"logs": [
    {
      "message": [
        "SyntaxError: Unexpected token 'd', \"data: {\"no\"... is not valid JSON"
      ],
      "level": "error",
      "timestamp": 1744242318653
    }
  ]

"logs": [
    {
      "message": [
        "SyntaxError: Unexpected token 'd', \"data: {\"no\"... is not valid JSON"
      ],
      "level": "error",
      "timestamp": 1744242318653
    }
  ]

"logs": [
    {
      "message": [
        "SyntaxError: Unexpected token 'd', \"data: {\"no\"... is not valid JSON"
      ],
      "level": "error",
      "timestamp": 1744242318653
    }
  ]

not sure what is wrong....

A

a4amaan•4/9/25, 11:53 PM

missing in autorage documentation...

Currently stream APIs (including official API and OpenAI compatible API) do not properly handle tool

Similar Threads

Similar Threads

Similar Threads