like this, "Output as valid json, dont write additional text or markdown"

R

Rapid•4/22/25, 11:48 AM

I have a function that I wanna call using LLM. So basically imagine a user writes I want to buy 5 NVDA stock.

So I want it to interpt to a JSON and then send the user a message with the parameters so he clicks "confirm" and then execute it. So basically just parsing text into predefined schema.

VViktor like this, "Output as valid json, dont write additional text or markdown"

R

Rapid•4/22/25, 12:03 PM

I will try it, was wondering if there is something that is meant for that though

RRapid I have a function that I wanna call using LLM. So basically imagine a user write...

V

ViktorOP•4/22/25, 12:37 PM

You can use functions / "tools" for that

V

ViktorOP•4/22/25, 12:39 PM

See this: https://developers.cloudflare.com/workers-ai/features/function-calling/
Basicly you enter your prompt, and pass additional functions that the LLM can call. If the LLM wants to call it, it'll pass information about the function, you execute the function, and add the result as

assistant

assistant

message.
There is a

cloudflare/ai-utils

cloudflare/ai-utils

making this simpler, but I'm struggling with that one right now (see my thread from sunday)

Cloudflare Docs

Function calling · Cloudflare Workers AI docs

Function calling enables people to take Large Language Models (LLMs) and use the model response to execute functions or interact with external APIs. The developer usually defines a set of functions and the required input schema for each function, which we call tools. The model then intelligently understands when it needs to do a tool call, and i...

V

ViktorOP•4/22/25, 12:42 PM

Also there is the "human in the loop", which can be used to actually intercept an ai workflow. But I didn't work with those ones yet, so I can't give any futher information if that is just a function waiting for confirmation, or has some other additional things
https://developers.cloudflare.com/agents/concepts/human-in-the-loop/

Cloudflare Docs

Human in the Loop · Cloudflare Agents docs

Human-in-the-Loop (HITL) workflows integrate human judgment and oversight into automated processes. These workflows pause at critical points for human review, validation, or decision-making before proceeding. This approach combines the efficiency of automation with human expertise and oversight where it matters most.

V

ViktorOP•4/22/25, 12:46 PM

Those are the issues I am having with ai-utils right now: #Is module @cloudflare/ai-utils stable?. If I don't get an answer here in the near future, I'll look for my access data for Github again and push it in as an issue, but so far it's not in a hurry and not a priority, so I'll wait here relaxed for an answer

R

Rapid•4/22/25, 1:15 PM

Thanks Viktor, I will check function-calling, as I said, I need to see the response before passing it to the function, as I want the user confirmation for the requested action. It just a faster possible way to interact with a bot, instead of clicking buttons and configuring settings

VViktor Also there is the "human in the loop", which can be used to actually intercept a...

R

Rapid•4/22/25, 1:16 PM

Thanks dude!

Z

zido•4/22/25, 6:02 PM

Can we add Molmo by Allenai please

Mmr.niko.la where is this playground the one i got doesnt have grok in it....

M

mr.niko.la•4/22/25, 7:09 PM

Ok I searched for this playground can’t fit in. I guess cf devs don’t follow this discord ?

C

Chris•4/22/25, 9:24 PM

Hi folks! What's the best way to do embedded function calling through an AI Gateway (e.g. to enable caching)?
https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/api-reference/

It looks like cloudflare/ai-utils doesn't let us pass in a

gateway

gateway

(as shown in the AI Gateway docs: https://developers.cloudflare.com/ai-gateway/providers/workersai/#workers-binding)

Though I'm not sure where to find detailed documentation on the AI binding — the docs don't say anything useful about the second parameter object: https://developers.cloudflare.com/workers-ai/configuration/bindings/#methods

The TypeScript types seem to have the best information. But still, it doesn't seem like AI binding / gateway /

runWithTools

runWithTools

are very composable.. or am I missing something?

J

jackoyam•4/22/25, 9:33 PM

Hi all, I keep getting AiError:5024 with @cf/meta/llama-3.1-8b-instruct-fast trying to use JSON Mode. Switching to @cf/meta/llama-4-scout-17b-16e-instruct works fine, but I'd like to use the cheaper model for certain tasks. I include this is the ai.run: response_format: { "type": "json_schema", "json_schema": schema },

Not sure why 3.1 keeps failing and whether there's an alternative?

Also: Docs are inconsistent with API Schema on each model's page, and the available models on the 'JSON Mode' page

J

JoaoCupido•4/23/25, 1:53 PM

Apparently, dreamshaper-8-lcm is giving an internal server error on my side. I don't know if it's only me or someone else has the same issue

VViktor AutoRAG does indexing when data is uploaded to R2. So if you need to re-index, m...

A

Alexander•4/24/25, 9:22 AM

Hey, thx for the tip - hmmm...... for some reason I may have thought otherwises. Isn't there a 4 hour window instead?

https://developers.cloudflare.com/autorag/configuration/indexing/

I will test it today to check but I was left with the impression that indexing is not on demand / on upload but on regular interval?

Cloudflare Docs

Indexing · AutoRAG

AutoRAG automatically indexes your data into vector embeddings optimized for semantic search. Once a data source is connected, indexing runs continuously in the background to keep your knowledge base fresh and queryable.

云

云•4/27/25, 1:26 AM

@cf/myshell-ai/melotts

@cf/myshell-ai/melotts

does not support chinese which the original model does support

A

akazwz•4/27/25, 7:55 AM

I find this problem too, will u support Chinese language ?

M

mounty•4/28/25, 10:25 AM

Is it possible to run HuggingFaceTB/SmolLM2-135M-Instruct on workersai?

Mmounty Is it possible to run HuggingFaceTB/SmolLM2-135M-Instruct on workersai?

I

Isaac McFadyen•4/28/25, 4:10 PM

?workers-ai-models TLDR: no.

S

SuperHelpflare•4/28/25, 4:10 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

B

ByteBeast•4/29/25, 5:14 AM

I have build an agent using agent package but I am stuck at one thing, I want to get URL params or query params in tools of MCP server like agentId so I dont have to pass agentId in tools props (paramsSchema).

N

nick•4/30/25, 4:20 AM

For the text-to-image models such as flux-1-schnell the current output is simply:

{ image: string }

{ image: string }

is there a way to have it output the usage data, similar to the

{ usage: { prompt_tokens: number, completion_tokens: number, total_tokens: number }}

{ usage: { prompt_tokens: number, completion_tokens: number, total_tokens: number }}

output for text generation models? Based on the pricing, I'd assume it would be something like

{ usage: { tiles: number, steps: number }}

{ usage: { tiles: number, steps: number }}

? Or does this just need to be calculated on each usage on our end using the default step of 4 if none is provided as input, and measuring the output image size?

Regarding output image size. Is there or will there be a way to specify the desired image output resolution? The image size i'm getting back is 1024x1024, so four 512x512 tiles. Just want to confirm this will always be the case?

Cloudflare Docs

flux-1-schnell

FLUX.1 [schnell] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions.

A

antim8•4/30/25, 12:54 PM

It seems like some models are not available in the worker types. For example https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct-fast/

or did i miss something?

Cloudflare Docs

llama-3.1-8b-instruct-fast

[Fast version] The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models. The Llama 3.1 instruction tuned text only models are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common indus...

A

antim8•4/30/25, 12:58 PM

those are available in the code. Seems like there are a bunch of model missing

R

Rabid•4/30/25, 7:12 PM

Anyone have a simple example for scoring NSFW images? I tried resnet-50 with a very NWFW image, and it just adds labels like "bath towel"

M

mr.niko.la•4/30/25, 8:35 PM

[ai]
binding = "AI"

[ai]
binding = "AI"

However, the

401 Unauthorized

401 Unauthorized

errors in my logs indicate that the authentication for Workers AI is failing, despite the correct binding. This suggests issues with account permissions, missing API tokens, or a misconfiguration in the Workers AI setup.

Do i need to add any token for

whisper turbo large v3

whisper turbo large v3

in the

vars

vars

section of my

wrangler.toml

wrangler.toml

file?

M

mr.niko.la•4/30/25, 8:50 PM

WORKERS_AI_TOKEN = "gdhd"

WORKERS_AI_TOKEN = "gdhd"

. i suppose i need to add the

workers ai

workers ai

api token too?

Mmr.niko.la `WORKERS_AI_TOKEN = "gdhd"`. i suppose i need to add the `workers ai` api token ...

I

Isaac McFadyen•4/30/25, 9:11 PM

You shouldn't need a token when you're using the binding, but how are you calling the model?

I

Isaac McFadyen•4/30/25, 9:11 PM

The binding is separate from the API, and if you're using the binding you shouldn't be calling the API at all

IIsaac McFadyen You shouldn't need a token when you're using the binding, but how are you callin...

M

mr.niko.la•4/30/25, 11:25 PM

Hi Isaac, thanks for the clarification! I'm using the AI binding to call the @cf/openai/whisper-large-v3-turbo model via env.AI.run in my Worker. Here’s a sample:

          try {
            console.log(`[BYPASS] Attempt 3: Using AI binding with raw bytes`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: [...audioBytes], // Convert Uint8Array to regular array
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 3: Using AI binding with raw bytes`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: [...audioBytes], // Convert Uint8Array to regular array
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 3: Using AI binding with raw bytes`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: [...audioBytes], // Convert Uint8Array to regular array
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 3: Using AI binding with raw bytes`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: [...audioBytes], // Convert Uint8Array to regular array
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 2: Using AI binding with base64`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: base64Audio,
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 2: Using AI binding with base64`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: base64Audio,
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 2: Using AI binding with base64`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: base64Audio,
              language: 'en',
            });

          try {
            console.log(`[BYPASS] Attempt 2: Using AI binding with base64`);
            const aiTranscription = await this.env.AI.run('@cf/openai/whisper-large-v3-turbo', {
              audio: base64Audio,
              language: 'en',
            });

Despite the binding in wrangler.toml ([ai] binding = "AI"), I’m getting 401 Unauthorized errors. Could this be an account permission issue or a binding misconfiguration? Any suggestions to debug?

Mmr.niko.la Hi Isaac, thanks for the clarification! I'm using the AI binding to call the @cf...

I

Isaac McFadyen•4/30/25, 11:36 PM

Is this in a Durable Object? Why are you using the

this

this

there?

IIsaac McFadyen Is this in a Durable Object? Why are you using the `this` there?

M

mr.niko.la•4/30/25, 11:56 PM

My

wrangler.toml

wrangler.toml

defines

AUDIO_PROCESSOR

AUDIO_PROCESSOR

as a Durable Object, and the code uses this.env.AI,

# Define the Durable Objects
[[durable_objects.bindings]]
name = "AUDIO_PROCESSOR"
class_name = "AudioProcessor"

# External DO binding for LLM worker
[[durable_objects.bindings]]
name = "LLM_PROCESSOR"
class_name = "LlmProcessor"
script_name = "llm"

# Add queue consumers with explicit handler functions
[[queues.consumers]]

[vars]


# Define the Durable Object class
[[migrations]]
tag = "v1"
new_classes = ["AudioProcessor"]

[[kv_namespaces]]


[[d1_databases]]


[[queues.consumers]]


[[queues.producers]]


[ai]
binding = "AI"

[observability]
enabled = true
head_sampling_rate = 1

# Define the Durable Objects
[[durable_objects.bindings]]
name = "AUDIO_PROCESSOR"
class_name = "AudioProcessor"

# External DO binding for LLM worker
[[durable_objects.bindings]]
name = "LLM_PROCESSOR"
class_name = "LlmProcessor"
script_name = "llm"

# Add queue consumers with explicit handler functions
[[queues.consumers]]

[vars]


# Define the Durable Object class
[[migrations]]
tag = "v1"
new_classes = ["AudioProcessor"]

[[kv_namespaces]]


[[d1_databases]]


[[queues.consumers]]


[[queues.producers]]


[ai]
binding = "AI"

[observability]
enabled = true
head_sampling_rate = 1

# Define the Durable Objects
[[durable_objects.bindings]]
name = "AUDIO_PROCESSOR"
class_name = "AudioProcessor"

# External DO binding for LLM worker
[[durable_objects.bindings]]
name = "LLM_PROCESSOR"
class_name = "LlmProcessor"
script_name = "llm"

# Add queue consumers with explicit handler functions
[[queues.consumers]]

[vars]


# Define the Durable Object class
[[migrations]]
tag = "v1"
new_classes = ["AudioProcessor"]

[[kv_namespaces]]


[[d1_databases]]


[[queues.consumers]]


[[queues.producers]]


[ai]
binding = "AI"

[observability]
enabled = true
head_sampling_rate = 1

# Define the Durable Objects
[[durable_objects.bindings]]
name = "AUDIO_PROCESSOR"
class_name = "AudioProcessor"

# External DO binding for LLM worker
[[durable_objects.bindings]]
name = "LLM_PROCESSOR"
class_name = "LlmProcessor"
script_name = "llm"

# Add queue consumers with explicit handler functions
[[queues.consumers]]

[vars]


# Define the Durable Object class
[[migrations]]
tag = "v1"
new_classes = ["AudioProcessor"]

[[kv_namespaces]]


[[d1_databases]]


[[queues.consumers]]


[[queues.producers]]


[ai]
binding = "AI"

[observability]
enabled = true
head_sampling_rate = 1

i removed the other bindings that were not relevant....

The sample code provided for Typsescipt uses the api only
https://developers.cloudflare.com/workers-ai/models/whisper-large-v3-turbo/

So im kinda confused and vibing use groq and claude to find the right way to do it thx

Cloudflare Docs

whisper-large-v3-turbo

Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation.

A

acusti•5/1/25, 2:19 AM

i’ve been using workers AI via the binding from my worker for many months and it‘s been great. overall, very happy with it, thanks to all who are putting in the work to make it work so well.

however, there is a frustrating DX paper cut, which is that when running the worker locally in dev, i have to authenticate at least once a day. it gets triggered by invoking

env.AI.run(…)

env.AI.run(…)

, and i get an authentication error in the console and it opens my default web browser to the cloudflare dashboard’s grant permissions UI (or sometimes crashes the process and i just run

wrangler login

wrangler login

manually to do it). is this normal and expected?

Mmr.niko.la My `wrangler.toml` defines `AUDIO_PROCESSOR` as a Durable Object, and the code u...

M

mr.niko.la•5/1/25, 6:57 AM

Capacity error, will retry: 3040: Capacity temporarily exceeded, please try again

Capacity error, will retry: 3040: Capacity temporarily exceeded, please try again

cf whisper worker ai. @Isaac | AS213339 is this common?

IIsaac McFadyen Is this in a Durable Object? Why are you using the `this` there?

M

mr.niko.la•5/1/25, 7:13 AM

export class AudioProcessor {
  constructor(state, env) {
    this.state = state;
    this.env = env;
  }

export class AudioProcessor {
  constructor(state, env) {
    this.state = state;
    this.env = env;
  }

export class AudioProcessor {
  constructor(state, env) {
    this.state = state;
    this.env = env;
  }

export class AudioProcessor {
  constructor(state, env) {
    this.state = state;
    this.env = env;
  }

T

TechOnyx•5/1/25, 8:19 AM

I recently wanted to integrate AI into my site, so I go into Cloudflare Dash > AI > Choose Workers API and create it, and then I deleted it immediately because I wanted to use the Rest API, but I can’t see that screen where you choose amongst worker and rest api, it keeps redirecting me to ai/usage

T

TechOnyx•5/1/25, 8:24 AM

Please help me, I'm locked out from the AI section, I'm unable to get my api keys and all that stuff

T

TechOnyx•5/1/25, 8:25 AM

I'm stuck on /usage page

R

rob•5/2/25, 2:44 PM

How do we know what models r the best for what use case. / modalities ? Is there like a matrix somewhere

Rrob How do we know what models r the best for what use case. / modalities ? Is there...

I

Isaac McFadyen•5/2/25, 8:26 PM

Just a matter of testing with your specific use-case

I

Isaac McFadyen•5/2/25, 8:26 PM

Specialty models like Qwen Coder will generally be better at their specific specialties than a non-specific model llike LLaMa

K

Keshav G•5/3/25, 8:08 AM

hello all
i am using runwithtools to build a rag chat app. the problem is the output message being outputted is very short, and the runwithtools doesn't respect the max_tokens. how to fix this ? kindly help

import { runWithTools } from '@cloudflare/ai-utils';
const model = '@cf/meta/llama-3.3-70b-instruct-fp8-fast';
const response = await runWithTools(
                    env.AI,
                    model,
                    {
                        messages: chatmessages,
                        tools: function_tools,
                        max_tokens: 8192,
                        temperature: 0.6,
                    },
                    { strictValidation: true, maxRecursiveToolRuns: 1, verbose: true, streamFinalResponse: true }
                );

import { runWithTools } from '@cloudflare/ai-utils';
const model = '@cf/meta/llama-3.3-70b-instruct-fp8-fast';
const response = await runWithTools(
                    env.AI,
                    model,
                    {
                        messages: chatmessages,
                        tools: function_tools,
                        max_tokens: 8192,
                        temperature: 0.6,
                    },
                    { strictValidation: true, maxRecursiveToolRuns: 1, verbose: true, streamFinalResponse: true }
                );

import { runWithTools } from '@cloudflare/ai-utils';
const model = '@cf/meta/llama-3.3-70b-instruct-fp8-fast';
const response = await runWithTools(
                    env.AI,
                    model,
                    {
                        messages: chatmessages,
                        tools: function_tools,
                        max_tokens: 8192,
                        temperature: 0.6,
                    },
                    { strictValidation: true, maxRecursiveToolRuns: 1, verbose: true, streamFinalResponse: true }
                );

import { runWithTools } from '@cloudflare/ai-utils';
const model = '@cf/meta/llama-3.3-70b-instruct-fp8-fast';
const response = await runWithTools(
                    env.AI,
                    model,
                    {
                        messages: chatmessages,
                        tools: function_tools,
                        max_tokens: 8192,
                        temperature: 0.6,
                    },
                    { strictValidation: true, maxRecursiveToolRuns: 1, verbose: true, streamFinalResponse: true }
                );

K

Keshav G•5/3/25, 2:23 PM

ok i sorted this issue, created a new package with these additional parameters.

now how can i get the final response and the previous tool response, to store as context for the ongoing conversation ?

Aacusti i’ve been using workers AI via the binding from my worker for many months and it...

V

ViktorOP•5/3/25, 5:52 PM

I logged in once, and that's it.
Sometimes it asks me which account I want to use (my login has multiple accounts connected to it), but that's like once in a week.

KKeshav G ok i sorted this issue, created a new package with these additional parameters. ...

V

ViktorOP•5/3/25, 6:01 PM

I honestly love everything about Cloudflare AI, except for their

ai-utils

ai-utils

ai-utils

ai-utils. I spend a lot of time troubleshooting it, it tracks token usage wrong (returns only the last ones), calls

AI.run

AI.run

AI.run

AI.run more often than necessary, etc. See #Is module @cloudflare/ai-utils stable? > I just wrote the function calling in my own code, works way better, and doesn't inflict double usage consumption. And I'd recommend you to do so, too, tbh

M

Mark•5/4/25, 2:54 AM

Does anyone still having index issue with AutoRag ?

MMark Does anyone still having index issue with AutoRag ?

I

iammac•5/4/25, 7:19 AM

Thread at #autorag

N

Nikhil Kachare•5/5/25, 9:59 AM

We recently transitioned from using the OpenAI Whisper API to Workers AI, using the @cf/openai/whisper-large-v3-turbo model. The experience has been excellent so far. One significant difference we've observed is with the language parameter: it defaults to 'en', unlike OpenAI, which automatically detects the language. Is this a feature missing from Workers AI, or am I overlooking something?

VViktor I honestly love everything about Cloudflare AI, **except** for their `ai-utils`....

K

Keshav G•5/5/25, 7:35 PM

yes, i have modified the function to return messages array for use in context later.
i have also published the changes as a npm package

K

Keshav G•5/5/25, 7:38 PM

i dont know, why the development is so ignored.. also the models are very weak
now way better open source models are there like the qwen, r1, even the llama 405b. cloudflare needs to up the game asap

A

Aiden•5/6/25, 7:23 AM

When to support qwen3

like this, "Output as valid json, dont write additional text or markdown"

Similar Threads

Similar Threads

Similar Threads