These are some stats of my setup: ``` [6:27:34 PM] read_form_data: 25.065ms [6:27:39 PM] transcribe

These are some stats of my setup:

[6:27:34 PM] read_form_data: 25.065ms
[6:27:39 PM] transcribe: 4.907s
[6:27:40 PM] generate_fist_text_chunk: 763.861ms
[6:27:41 PM] generate_speech_chunk_0: 1.476s
[6:27:42 PM] generate_speech_chunk_1: 901.312ms


[6:28:02 PM] read_form_data: 19.01ms
[6:28:06 PM] transcribe: 3.367s
[6:28:06 PM] generate_fist_text_chunk: 612.575ms
[6:28:07 PM] generate_speech_chunk_0: 1.148s
[6:28:09 PM] generate_speech_chunk_1: 1.428s
[6:28:10 PM] generate_speech_chunk_2: 1.582s
[6:28:12 PM] generate_speech_chunk_3: 1.267s


[6:28:26 PM] read_form_data: 5.596ms
[6:28:27 PM] transcribe: 1.593s
[6:28:28 PM] generate_fist_text_chunk: 798.88ms
[6:28:29 PM] generate_speech_chunk_0: 1.294s
[6:28:30 PM] generate_speech_chunk_1: 698.474ms
[6:28:31 PM] generate_speech_chunk_2: 920.976ms

[6:27:34 PM] read_form_data: 25.065ms
[6:27:39 PM] transcribe: 4.907s
[6:27:40 PM] generate_fist_text_chunk: 763.861ms
[6:27:41 PM] generate_speech_chunk_0: 1.476s
[6:27:42 PM] generate_speech_chunk_1: 901.312ms


[6:28:02 PM] read_form_data: 19.01ms
[6:28:06 PM] transcribe: 3.367s
[6:28:06 PM] generate_fist_text_chunk: 612.575ms
[6:28:07 PM] generate_speech_chunk_0: 1.148s
[6:28:09 PM] generate_speech_chunk_1: 1.428s
[6:28:10 PM] generate_speech_chunk_2: 1.582s
[6:28:12 PM] generate_speech_chunk_3: 1.267s


[6:28:26 PM] read_form_data: 5.596ms
[6:28:27 PM] transcribe: 1.593s
[6:28:28 PM] generate_fist_text_chunk: 798.88ms
[6:28:29 PM] generate_speech_chunk_0: 1.294s
[6:28:30 PM] generate_speech_chunk_1: 698.474ms
[6:28:31 PM] generate_speech_chunk_2: 920.976ms

[6:27:34 PM] read_form_data: 25.065ms
[6:27:39 PM] transcribe: 4.907s
[6:27:40 PM] generate_fist_text_chunk: 763.861ms
[6:27:41 PM] generate_speech_chunk_0: 1.476s
[6:27:42 PM] generate_speech_chunk_1: 901.312ms


[6:28:02 PM] read_form_data: 19.01ms
[6:28:06 PM] transcribe: 3.367s
[6:28:06 PM] generate_fist_text_chunk: 612.575ms
[6:28:07 PM] generate_speech_chunk_0: 1.148s
[6:28:09 PM] generate_speech_chunk_1: 1.428s
[6:28:10 PM] generate_speech_chunk_2: 1.582s
[6:28:12 PM] generate_speech_chunk_3: 1.267s


[6:28:26 PM] read_form_data: 5.596ms
[6:28:27 PM] transcribe: 1.593s
[6:28:28 PM] generate_fist_text_chunk: 798.88ms
[6:28:29 PM] generate_speech_chunk_0: 1.294s
[6:28:30 PM] generate_speech_chunk_1: 698.474ms
[6:28:31 PM] generate_speech_chunk_2: 920.976ms

[6:27:34 PM] read_form_data: 25.065ms
[6:27:39 PM] transcribe: 4.907s
[6:27:40 PM] generate_fist_text_chunk: 763.861ms
[6:27:41 PM] generate_speech_chunk_0: 1.476s
[6:27:42 PM] generate_speech_chunk_1: 901.312ms


[6:28:02 PM] read_form_data: 19.01ms
[6:28:06 PM] transcribe: 3.367s
[6:28:06 PM] generate_fist_text_chunk: 612.575ms
[6:28:07 PM] generate_speech_chunk_0: 1.148s
[6:28:09 PM] generate_speech_chunk_1: 1.428s
[6:28:10 PM] generate_speech_chunk_2: 1.582s
[6:28:12 PM] generate_speech_chunk_3: 1.267s


[6:28:26 PM] read_form_data: 5.596ms
[6:28:27 PM] transcribe: 1.593s
[6:28:28 PM] generate_fist_text_chunk: 798.88ms
[6:28:29 PM] generate_speech_chunk_0: 1.294s
[6:28:30 PM] generate_speech_chunk_1: 698.474ms
[6:28:31 PM] generate_speech_chunk_2: 920.976ms

cosbgnOP•7/25/25, 8:43 PM

transcribe is the one which takes the most. (These are using openai endpoints)

cosbgnOP•7/25/25, 8:44 PM

After generate_speech_chunk_0generate_speech_chunk_0 that is already sent back to the client, so the other chunks don't really matter

cosbgnOP•7/25/25, 8:44 PM

Ideally I would like to get transcribe to a few hundreds of milliseconds, not 5 seconds.

Ccosbgn These are some stats of my setup: ``` [6:27:34 PM] read_form_data: 25.065ms [6:...

Isaac McFadyen•7/25/25, 8:47 PM

How long is your audio?

cosbgnOP•7/25/25, 8:47 PM

a sentence

cosbgnOP•7/25/25, 8:47 PM

couple of seconds, like 10 seconds maybe

creet•7/26/25, 5:14 AM

Hmm yeah 2x transcription speed is pretty bad

creet•7/26/25, 5:14 AM

Make sure you’re using whisper v3 large turbo or tiny en

creet•7/26/25, 5:15 AM

Not the base whisper

zsueo•7/29/25, 4:33 PM

hey im using llama-4-scout-17b-16e-instruct for tool calling and it seems to just get stuck in a call loop? like it just keeps calling the tool over and over

zsueo•7/29/25, 6:54 PM

it seems to be doing this for both this and the hermes model

zsueo•7/29/25, 6:54 PM

im sending in requests like this

{
  "messages": [
    {
      "role": "tool",
      "tool_call_id": "chatcmpl-tool-04bad465677b4783b6797433a8deb05b",
      "content": "Pretend you got databack this tool is still under construction"
    },
    {
      "role": "tool",
      "tool_call_id": "chatcmpl-tool-5d73b086fe964ae2900cdd527a659f1f",
      "content": "Pretend you got databack this tool is still under construction"
    },
    {
      "role": "tool",
      "tool_call_id": "chatcmpl-tool-0140345ec2584ae0b8d4d6d1e3c6585e",
      "content": "Pretend you got databack this tool is still under construction"
    },
    {
      "role": "user",
      "content": "hey can you fetch data relating to joe?"
    },
    {
      "role": "system",
      "content": "You are a database assistant, you will query the database if a user requests information from it and respond with a human readable output, please only call the tool once and respond each time after a tool call, all tool responses have the role tool"
    }
  ],
  "max_tokens": 1024,
  "raw": false,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "Query Vector Database",
        "description": "Queries the Vector Database for legal documents matching the provided query",
        "parameters": {
          "type": "object",
          "properties": {
            "Query": {
              "type": "string",
              "description": "The query to the Vector db"
            }
          },
          "required": ["Query"]
        }
      }
    }
  ]
}

{
  "messages": [
    {
      "role": "tool",
      "tool_call_id": "chatcmpl-tool-04bad465677b4783b6797433a8deb05b",
      "content": "Pretend you got databack this tool is still under construction"
    },
    {
      "role": "tool",
      "tool_call_id": "chatcmpl-tool-5d73b086fe964ae2900cdd527a659f1f",
      "content": "Pretend you got databack this tool is still under construction"
    },
    {
      "role": "tool",
      "tool_call_id": "chatcmpl-tool-0140345ec2584ae0b8d4d6d1e3c6585e",
      "content": "Pretend you got databack this tool is still under construction"
    },
    {
      "role": "user",
      "content": "hey can you fetch data relating to joe?"
    },
    {
      "role": "system",
      "content": "You are a database assistant, you will query the database if a user requests information from it and respond with a human readable output, please only call the tool once and respond each time after a tool call, all tool responses have the role tool"
    }
  ],
  "max_tokens": 1024,
  "raw": false,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "Query Vector Database",
        "description": "Queries the Vector Database for legal documents matching the provided query",
        "parameters": {
          "type": "object",
          "properties": {
            "Query": {
              "type": "string",
              "description": "The query to the Vector db"
            }
          },
          "required": ["Query"]
        }
      }
    }
  ]
}

but it keeps trying to call the tool again

zsueo•7/29/25, 7:16 PM

do i need to send the assistant querying the tool as part of the message history?

DrWayne•7/29/25, 7:25 PM

I'm having trouble with the https://api.cloudflare.com/client/v4/accounts/{accountId}/ai/models/search endpoint in-browser. The fetch triggers an OPTIONS request that fails with "400 Bad Request".
The GET request works just fine via curl on the command line, and the OPTIONS request curl returns
{"success":false,"errors":[{"code":9106,"message":"Missing X-Auth-Key, X-Auth-Email or Authorization headers"}]}{"success":false,"errors":[{"code":9106,"message":"Missing X-Auth-Key, X-Auth-Email or Authorization headers"}]}

Since the OPTIONS call is checking whether the endpoint allows those headers, it will never include them in the request. I'm betting it is just an overzealous config on needing the auth header/s missing an exception for the OPTIONS call. Or am I missing something here?

Zzsueo do i need to send the assistant querying the tool as part of the message history...

Isaac McFadyen•7/29/25, 7:38 PM

Yes, both the tool result and the tool call need to be included as a message.

Isaac McFadyen•7/29/25, 7:38 PM

Also it looks like you have the messages array backwards. Lower in the array (higher index) is more recent.

Isaac McFadyen•7/29/25, 7:38 PM

I.e. system, then user, then assistant tool call, then tool result.

IIsaac McFadyen Also it looks like you have the messages array backwards. Lower in the array (hi...

zsueo•7/29/25, 7:38 PM

huh the example had it higher in the array as first, easier for me to manage

IIsaac McFadyen Yes, both the tool result and the tool call need to be included as a message.

zsueo•7/29/25, 7:39 PM

the input schema for this model doesnt have anything like that i guess the docs are a bit behind

Zzsueo huh the example had it higher in the array as first, easier for me to manage

zsueo•7/29/25, 7:39 PM

https://github.com/craigsdennis/lightbulb-moment-tool-calling/blob/main/src/index.ts <- this is the example i was referencing

GitHub

lightbulb-moment-tool-calling/src/index.ts at main · craigsdennis/...

Workers AI Tool Calling Example. Contribute to craigsdennis/lightbulb-moment-tool-calling development by creating an account on GitHub.

Zzsueo https://github.com/craigsdennis/lightbulb-moment-tool-calling/blob/main/src/inde...

Isaac McFadyen•7/29/25, 7:41 PM

Yeah, so the unshiftunshift adds the system message to the start of the array

zsueo•7/29/25, 7:42 PM

oh right myb i got it mixed up

IIsaac McFadyen Yes, both the tool result and the tool call need to be included as a message.

zsueo•7/29/25, 7:42 PM

is there a recommended schema for this?

zsueo•7/29/25, 7:42 PM

i cant find anything on cloudflares docs

Isaac McFadyen•7/29/25, 7:45 PM

I'm not 100% sure, but I expect it mirrors the OpenAI schema, so it would be an assistant message but with tool_callstool_calls instead of string

content

content

: https://platform.openai.com/docs/api-reference/chat/create

Isaac McFadyen•7/29/25, 7:45 PM

Worth a try, if it's wrong you'll get an error that might help troubleshoot further

zsueo•7/29/25, 7:46 PM

thanks so much!

IIsaac McFadyen I'm not 100% sure, but I expect it mirrors the OpenAI schema, so it would be an ...

zsueo•7/29/25, 9:02 PM

it worked!! just incase anyone wants to do something similar this is the schema i used

   {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "039920ecd55b4b5ba689d692933602cc",
          "function": {
            "name": "Query Vector Database",
            "arguments": "{\"Query\": \"joe\"}"
          },
          "type": "function"
        }
      ],
      "content": ""
    }

   {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "039920ecd55b4b5ba689d692933602cc",
          "function": {
            "name": "Query Vector Database",
            "arguments": "{\"Query\": \"joe\"}"
          },
          "type": "function"
        }
      ],
      "content": ""
    }

zsueo•7/30/25, 6:02 PM

hey im noticing that sometimes the ai does not call the tool properly leading to json just floating in the reply, what is the expected format the ai is supposed to output in so i can try and force that?

ed•7/30/25, 7:54 PM

when's the next models update? would love to have kimi-k2, GLM, etc

rob•7/31/25, 6:16 PM

apparently maybe K2 soon

https://x.com/_mchenco/status/1947761683339825207

michelle (@_mchenco)

i ate my words. who's got a use case for kimi on workers ai? let's chat

•

7/22/25, 8:52 PM

Shadi•8/1/25, 6:09 AM

i kindly pray by my bedside that we support allminiLmv6allminiLmv6 in cloudflare workers ai.

would acc be so insane.

im using it for my current embeddings and i don't think its best practice to use 2 different embedding models, as I see we only support bge.

Shadi•8/1/25, 4:35 PM

lmao

R3AP3R editz•8/3/25, 3:06 PM

can anyone say how can we change the embdding model in autorag?
and also does autorag support multilanguae?

ed•8/4/25, 3:57 PM

✘ [ERROR] Uncaught (in promise) InferenceUpstreamError: 10000: Authentication error

      at _parseError (cloudflare-internal:ai-api:107:24)

✘ [ERROR] Uncaught (in promise) InferenceUpstreamError: 10000: Authentication error

      at _parseError (cloudflare-internal:ai-api:107:24)

Anyone else getting this error? Tried logging in and out over and over again

iqorex•8/5/25, 7:09 PM

Is there any chance we will see this model soon in workers ai

? https://openai.com/index/introducing-gpt-oss/

Introducing gpt-oss

gpt-oss-120b and gpt-oss-20b push the frontier of open-weight reasoning models

Original message was deleted

Isaac McFadyen•8/5/25, 9:15 PM

It's definitely not 0 :lul:
https://blog.cloudflare.com/openai-gpt-oss-on-workers-ai/

The Cloudflare Blog

Partnering with OpenAI to bring their new open models onto Cloudfla...

OpenAI’s newest open-source models are now available on Cloudflare Workers AI on Day 0, with support for Responses API, Code Interpreter and Web Search (coming soon).

iqorex•8/6/25, 8:36 AM

damn that was fast

�

𝗞enneth•8/6/25, 6:03 PM

Hi Cloudflare team,

Does the workers ai @cf/openai/gpt-oss-120b model support structured outputs with JSON schemas?

I need to force the model to return responses in a specific JSON format using response_format parameter. I don't see this mentioned in the docs.

�𝗞enneth Hi Cloudflare team, Does the workers ai @cf/openai/gpt-oss-120b model support ...

Chaika•8/6/25, 6:19 PM

They're using the new responses openai format:
https://platform.openai.com/docs/api-reference/responses/create

which supports that under the text object looks like. Tested it quickly and it seems to work fine for me

�

𝗞enneth•8/6/25, 6:22 PM

thanks @Chaika

akazwz•8/7/25, 10:26 AM

will gpt-oss-120b support stream ?

Aakazwz will gpt-oss-120b support stream ?

Chaika•8/7/25, 3:37 PM

It already does support streaming via openai's new response format streaming:
https://platform.openai.com/docs/guides/streaming-responses?api-mode=responses

akazwz•8/7/25, 5:00 PM

I was using the response api. without stream param, it works, with stream: true, server error, return 500

akazwz•8/7/25, 5:00 PM

Aakazwz I was using the response api. without stream param, it works, with stream: true,...

Chaika•8/7/25, 5:20 PM

Have to use Workers binding for streaming

Chaika•8/7/25, 5:23 PM

I don't think the direct api has supported streaming for any model? Not 100% sure as I don't use the api myself, but yea that same request via binding streaming looks fine

Dev2xl•8/9/25, 10:35 PM

Hello,
I’m exploring options for running inference on private TensorFlow models with Cloudflare’s products. Is it possible to achieve this using Workers or any other Cloudflare service? Any technical guidance or references would be greatly appreciated. Thank you!

Isaac McFadyen•8/10/25, 4:16 PM

You could try Containers but they're currently ill-suited for most ML things as they have fairly low resource allocation and no GPUs.

ixixix•8/12/25, 6:44 AM

I know the docs don't mention it, but just to confirm, the /audio/transcriptions endpoint isn't supported as an openai compatible endpoint in Workers AI or AI Gateway for the cloudflare whisper models, right? I'm getting {"errors":[{"code":7000,"message":"No route for that URI"}]}{"errors":[{"code":7000,"message":"No route for that URI"}]} and {"code":2002,"message":"Internal server error"}{"code":2002,"message":"Internal server error"}

These are some stats of my setup: ``` [6:27:34 PM] read_form_data: 25.065ms [6:27:39 PM] transcribe

Similar Threads

Similar Threads

Similar Threads