Is module @cloudflare/ai-utils stable?

Original message was deleted

Viktor•4/19/25, 5:13 PM

Hey everyone,

I am currently evaluating whether we should integrate Cloudflare as a provider for AI for our software. A large part of this will be retrieving data, which will then be further processed via AI.
The functions that can be provided are ideal for this. I'm currently looking at @cloudflare/ai-utils@cloudflare/ai-utils.

In the (minimally) modified code from the fetch() example, no token consumption is returned to me. The example is as follows:
https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/examples/fetch/
I attach the source code of my changes, but they are almost identical to the original.

If I change the model (without changing the source code in any other way), for example llama-3.3-70b-instruct-fp8-fastllama-3.3-70b-instruct-fp8-fast, functions are executed according to the console, but not included in the response. The same applies to @cf/meta/llama-4-scout-17b-16e-instruct@cf/meta/llama-4-scout-17b-16e-instruct. However, the consumption is then returned correctly.

When I look at the source code, it looks to me as a layman as if streamFinalResponse = truestreamFinalResponse = true would first execute AI.runAI.run without stream, and if streamFinalResponse = truestreamFinalResponse = true is then set, it would be executed again with stream.
Am I missing something? Doesn't this cause double calculation of input/output tokens AND pass wrong consumption data (as only those from Cloudflare reported are passed, and the one from the previous call were ignored)?
First invoke of function: https://github.com/cloudflare/ai-utils/blob/2ac3312bef885976b7bae32da761a58eeb896735/src/runWithTools.ts#L126C1-L136C6
Rerun if no tool call and proper output: https://github.com/cloudflare/ai-utils/blob/2ac3312bef885976b7bae32da761a58eeb896735/src/runWithTools.ts#L237

Might also be possible that I oversaw some "work in progress" hint somewhere. But as far as it looks right now the module (a) returns always wrong (consumption) data, or (b) does not work properly with every model

Thank you in advance for any clarifications!

example-with-llama-4.js2.64KB

example-with-different-model.js2.61KB

example-gives-0-token-consumption.js1.86KB

Cloudflare Docs

Use fetch() handler · Cloudflare Workers AI docs

Learn how to use the fetch() handler in Cloudflare Workers AI to enable LLMs to perform API calls, like retrieving a 5-day weather forecast using function calling.

GitHub

ai-utils/src/runWithTools.ts at 2ac3312bef885976b7bae32da761a58eeb8...

Developer toolkit that makes it simple to build with the Workers AI platform. - cloudflare/ai-utils

Viktor•4/23/25, 7:16 PM

I somehow managed it by myself by writing my own function, that
(a) starts reading with stream = true
(b) checks if the output seems to be a tool call (by buffering the response, and checking with buffer.startsWith("<tool_call")buffer.startsWith("<tool_call")
(c) if fetches everything, executes the tool and tracks the consumption, and reruns the code
(d) or if it's not a tool call recreates a new stream and pipes everything in it - including old consumption

It's ugly code and only working for streaming, but it's working for now.
As it's ugly, very specific, and I don't have my github credentials (yet) I won't issue a pull request to <@896061789899485195>/ai-utils<@896061789899485195>/ai-utils

As sidenote: when using different models the output for tool calls vary while streaming
E.g. llama has raw json, mistral has json wrapped in xml tags.

Is module @cloudflare/ai-utils stable?

Similar Threads

Similar Threads

Similar Threads