Does anyone have any view on which is the best text genertion model that reliably outputs JSON? I'm

MMitya Does anyone have any view on which is the best text genertion model that reliabl...

I

Isaac McFadyen•2/5/25, 6:28 PM

Your best bet is to implement retries, really.

I

Isaac McFadyen•2/5/25, 6:28 PM

Try parsing the JSON and if it fails re-send the request.

I

Isaac McFadyen•2/5/25, 6:28 PM

Unless specifically forced to output JSON via low-level logit math (which Workers AI doesn't support, I don't think) then there will always be some cases where it returns invalid JSON.

I

Isaac McFadyen•2/5/25, 6:29 PM

You can try to improve it by using specific system messages but it'll never be 100% without forced logits.

IIsaac McFadyen Your best bet is to implement retries, really.

M

MityaOP•2/6/25, 9:27 AM

Thanks. Yeah this is what I'm doing currently. It's helping a lot, but as you say, there's always some cases. I note that the mistral model (CF's single function calls-capable model, at least according to the filters on the models page) has a "JSON mode". But in the input params, this is not mentioned, so supposedly there's a way to force it to return JSON but I'm unsure whether there's a param missing from the docs.

M

MityaOP•2/6/25, 9:28 AM

Also, this looks interesting re: structured output: https://js.langchain.com/docs/how_to/structured_output/

How to return structured data from a model | 🦜️🔗 Langchain

It is often useful to have a model return output that matches some

MMitya Thanks. Yeah this is what I'm doing currently. It's helping a lot, but as you sa...

I

Isaac McFadyen•2/6/25, 1:01 PM

I don't believe there is. That's for when you're running it locally.

MMitya Thanks. Yeah this is what I'm doing currently. It's helping a lot, but as you sa...

T

thomash•2/6/25, 2:04 PM

if you are using the OpenAI compatible API, I think you can enable JSON mode by passing the correct

response_format

response_format

response_format

response_format. Since it is possible to access of CloudFlare's models via the OpenAI API.

Tthomash if you are using the OpenAI compatible API, I think you can enable JSON mode by ...

I

Isaac McFadyen•2/6/25, 2:15 PM

Cloudflare's API does not respect the JSON mode setting iirc

I

Isaac McFadyen•2/6/25, 2:15 PM

I tried the other day and it ignored it.

T

Torianna•2/6/25, 2:16 PM

to be fair, json mode is only supported on a small amount of models

T

Torianna•2/6/25, 2:16 PM

since it requires modifying how the model responds

TTorianna to be fair, json mode is only supported on a small amount of models

I

Isaac McFadyen•2/6/25, 2:17 PM

Kind of? Like, the models themselves have "JSON mode" but it's not actually that hard to force it to be JSON, it just needs some forced logits but theoretically any backend that does that will work with any model.

I

Isaac McFadyen•2/6/25, 2:17 PM

I just don't think CF's backend has that setting or if it does it's not enabled.

T

Torianna•2/6/25, 2:18 PM

true

T

Torianna•2/6/25, 2:18 PM

but also looking at the models CF has, most dont allow modifying the logit bias

M

Muhammadreza Haghiri•2/6/25, 2:24 PM

Hi everyone.
I am using CF AI workers a lot and recently, my startup (basically me and my team) relesead this new 8B multilingual model:

https://huggingface.co/mann-e/Hormoz-8B

I just want to know how can we use it through AI workers?

mann-e/Hormoz-8B · Hugging Face

MMuhammadreza Haghiri Hi everyone. I am using CF AI workers a lot and recently, my startup (basically...

I

Isaac McFadyen•2/6/25, 2:32 PM

?workers-ai-models

S

SuperHelpflare•2/6/25, 2:32 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

M

MityaOP•2/6/25, 4:29 PM

Update on the structured/JSON output discussion above, I've had great results with a mixture of LangChain, native JSON schemas and Mistral (when using Mistral directly via their own API). Here's a truncated example:

import { ChatMistralAI } from "@langchain/mistralai"
router.get('/mistral', async req => {
  const model = new ChatMistralAI({
    model: "mistral-large-latest",
    temperature: 0,
    apiKey: <MISTRAL-API-KEY>
  });
  const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>);
  return await structuredLlm.invoke(<YOUR-PROMPT-HERE>);
})

import { ChatMistralAI } from "@langchain/mistralai"
router.get('/mistral', async req => {
  const model = new ChatMistralAI({
    model: "mistral-large-latest",
    temperature: 0,
    apiKey: <MISTRAL-API-KEY>
  });
  const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>);
  return await structuredLlm.invoke(<YOUR-PROMPT-HERE>);
})

import { ChatMistralAI } from "@langchain/mistralai"
router.get('/mistral', async req => {
  const model = new ChatMistralAI({
    model: "mistral-large-latest",
    temperature: 0,
    apiKey: <MISTRAL-API-KEY>
  });
  const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>);
  return await structuredLlm.invoke(<YOUR-PROMPT-HERE>);
})

import { ChatMistralAI } from "@langchain/mistralai"
router.get('/mistral', async req => {
  const model = new ChatMistralAI({
    model: "mistral-large-latest",
    temperature: 0,
    apiKey: <MISTRAL-API-KEY>
  });
  const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>);
  return await structuredLlm.invoke(<YOUR-PROMPT-HERE>);
})

Returns JSON nearly every time and there's support for built-in retries where it fails.

C

crazyjack12•2/6/25, 7:11 PM

There was a beta to be able to use a hugging face model, but not sure what’s the state of that project @Muhammadreza Haghiri

D

dandoen•2/7/25, 11:44 AM

maybe this channel is more appropriate than vectorize, but i'm wondering if there are any short term plans to make the multi-lingual text embedding model BAAI/bge-m3 available? current BGE ones are EN only
vectorize

G

GaryLi•2/7/25, 4:11 PM

asking if workers ai can get full-sized deepseek r1 model

.

.triloksingh•2/7/25, 4:48 PM

#workers-aiI have a paid plan, but I want to make expectations for my admin panel. my website url is example.com/admin

P

prisum 🎄•2/8/25, 1:01 AM

so if I upgrade to the $5 a month plan I get infinite ai requests?

Pprisum 🎄so if I upgrade to the $5 a month plan I get infinite ai requests?

G

GaryLi•2/8/25, 10:42 AM

go to docs, theres s pricing list for ai

P

Puliczek•2/8/25, 12:36 PM

what type of MCP is used by worker-mcp? command or SSE?

N

nukely•2/8/25, 5:37 PM

any tutorial on how to use Gemini Multimodal Live API with nextjs and a websocket server

GGaryLi go to docs, theres s pricing list for ai

P

prisum 🎄•2/8/25, 6:49 PM

Yeah I did but on the direct worker ai page it says 30,000 requests monthly but on the pricing document it says unlimited

Pprisum 🎄Yeah I did but on the direct worker ai page it says 30,000 requests monthly but ...

N

Nutjob 홍•2/8/25, 10:18 PM

Which page are you looking at?

https://developers.cloudflare.com/workers-ai/platform/pricing/

10,000 free tokens a day and then pricing based on model size and blended input/output tokens.

Cloudflare Docs

Pricing · Cloudflare Workers AI docs

Workers AI is included in both the Free and Paid Workers plans and is priced based on model task, model size, and units.

Eelement14 I would also like to know if there is any REST API Endpoint (or via wrangler com...

U

user•2/9/25, 7:59 AM

Anything on this?

M

Muhammadreza Haghiri•2/9/25, 3:57 PM

Will FLUX-Dev be added to the models as well?

S

shoo•2/9/25, 5:09 PM

InferenceUpstreamError: undefined: undefined

anyone who knows workaorund to this?

if I use --remote, it doesn't connect to any other worker in local as workers binding

B

Bob•2/9/25, 7:23 PM

Hello. I have a question.
bge-large-en-v1.5 <- for this model, how many text items I can send in 1 request? I can't seem to find any info on it.

BBumblebeeSquare hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does...

B

BumblebeeSquare•2/10/25, 9:18 PM

L

leisure•2/11/25, 6:19 AM

Hi, I have a question. When I use it, I find that Daily Usage Limits will not reset at zero normally. Is this a special problem?

BBumblebeeSquare Empty message

R

Raylight•2/11/25, 11:27 AM

Yes. It's not implemented in the package. I created a quick fix (https://github.com/Raylight-PWL/workers-ai-provider/pull/1) but haven't used/tested the package. In any case, you can look at the diff if you want to patch it yourself.

GGaryLi asking if workers ai can get full-sized deepseek r1 model

G

GaryLi•2/11/25, 3:34 PM

idk why they dont put this on

R

rob•2/11/25, 3:48 PM

i dont think they can handle the size

R

rob•2/11/25, 3:48 PM

yet

M

Marco Aleixo•2/11/25, 7:51 PM

do you plan to add more embedding models?
only having a single not multi lingual model available for embedding is kind of a big blocker to use workers ai

B

beingmudit [WePlayDOS]•2/12/25, 9:08 PM

Hi CF team, had a doubt about function calling and the supported model. I can see that only one model is supported but in the documentation of this llama model - https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct/

I can see in the input/output params, we can pass tools. Is there a documentation mismatch?

Cloudflare Docs

llama-3.2-11b-vision-instruct · Cloudflare Workers AI docs

The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image.

B

beingmudit [WePlayDOS]•2/12/25, 9:48 PM

One more q - I am trying out the hermes model and sending this payload as per this example - https://developers.cloudflare.com/workers-ai/function-calling/traditional/

I am using the gateway to make the call - https://gateway.ai.cloudflare.com/v1/{ACC_ID}/{gateway}/workers-ai/v1/chat/completions

const response = await env.AI.run("@hf/nousresearch/hermes-2-pro-mistral-7b", {
  messages: [
    {
      role: "user",
      content: "what is the weather in london?",
    },
  ],
  tools: [
    {
      name: "getWeather",
      description: "Return the weather for a latitude and longitude",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
            description: "The latitude for the given location",
          },
          longitude: {
            type: "string",
            description: "The longitude for the given location",
          },
        },
        required: ["latitude", "longitude"],
      },
    },
  ],
});

return new Response(JSON.stringify(response.tool_calls));

const response = await env.AI.run("@hf/nousresearch/hermes-2-pro-mistral-7b", {
  messages: [
    {
      role: "user",
      content: "what is the weather in london?",
    },
  ],
  tools: [
    {
      name: "getWeather",
      description: "Return the weather for a latitude and longitude",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
            description: "The latitude for the given location",
          },
          longitude: {
            type: "string",
            description: "The longitude for the given location",
          },
        },
        required: ["latitude", "longitude"],
      },
    },
  ],
});

return new Response(JSON.stringify(response.tool_calls));

const response = await env.AI.run("@hf/nousresearch/hermes-2-pro-mistral-7b", {
  messages: [
    {
      role: "user",
      content: "what is the weather in london?",
    },
  ],
  tools: [
    {
      name: "getWeather",
      description: "Return the weather for a latitude and longitude",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
            description: "The latitude for the given location",
          },
          longitude: {
            type: "string",
            description: "The longitude for the given location",
          },
        },
        required: ["latitude", "longitude"],
      },
    },
  ],
});

return new Response(JSON.stringify(response.tool_calls));

const response = await env.AI.run("@hf/nousresearch/hermes-2-pro-mistral-7b", {
  messages: [
    {
      role: "user",
      content: "what is the weather in london?",
    },
  ],
  tools: [
    {
      name: "getWeather",
      description: "Return the weather for a latitude and longitude",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
            description: "The latitude for the given location",
          },
          longitude: {
            type: "string",
            description: "The longitude for the given location",
          },
        },
        required: ["latitude", "longitude"],
      },
    },
  ],
});

return new Response(JSON.stringify(response.tool_calls));

I am getting back this

{
  "id": "id-1739396566250",
  "object": "chat.completion",
  "created": 1739396566,
  "model": "@hf/nousresearch/hermes-2-pro-mistral-7b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "input": 15,
    "output": 0,
    "total": 15
  }
}

{
  "id": "id-1739396566250",
  "object": "chat.completion",
  "created": 1739396566,
  "model": "@hf/nousresearch/hermes-2-pro-mistral-7b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "input": 15,
    "output": 0,
    "total": 15
  }
}

{
  "id": "id-1739396566250",
  "object": "chat.completion",
  "created": 1739396566,
  "model": "@hf/nousresearch/hermes-2-pro-mistral-7b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "input": 15,
    "output": 0,
    "total": 15
  }
}

{
  "id": "id-1739396566250",
  "object": "chat.completion",
  "created": 1739396566,
  "model": "@hf/nousresearch/hermes-2-pro-mistral-7b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "input": 15,
    "output": 0,
    "total": 15
  }
}

Seems like function calling is not working. Can anyone help?

Cloudflare Docs

Traditional function calling · Cloudflare Workers AI docs

This page shows how you can do traditional function calling, as defined by industry standards. Workers AI also offers embedded function calling, which is drastically easier than traditional function calling.

J

Jason Barry•2/12/25, 9:52 PM

yes, my function calling isn't working either, using

@cf/meta/llama-3.1-8b-instruct-fast

@cf/meta/llama-3.1-8b-instruct-fast

@cf/meta/llama-3.1-8b-instruct-fast

@cf/meta/llama-3.1-8b-instruct-fast. I'm seeing a lot of

Error: oneOf at '.' not met: required properties at '.name' is 'type,description', required properties at '.tools.0.function.parameters' is 'type,properties'

Error: oneOf at '.' not met: required properties at '.name' is 'type,description', required properties at '.tools.0.function.parameters' is 'type,properties'

Error: oneOf at '.' not met: required properties at '.name' is 'type,description', required properties at '.tools.0.function.parameters' is 'type,properties'

Error: oneOf at '.' not met: required properties at '.name' is 'type,description', required properties at '.tools.0.function.parameters' is 'type,properties'

errors in my AI Gateway logs. I'm comparing to requests that I sent last friday -- the request schema I'm using is the same, but something is causing it to error out

J

Jason Barry•2/12/25, 9:55 PM

internalCode 5006, fwiw

J

Jason Barry•2/12/25, 9:57 PM

looking in my logs I see the error starting last night, here's an example payload that used to work that is currently failing

{
  "prompt": "Find the computer specs",
  "temperature": 0,
  "messages": [
    {
      "role": "user",
      "content": "..."
    }
  ],
  "tools": [
    {
      "name": "dataExtractor",
      "description": "Extract data that conforms to the provided schema",
      "parameters": {
        "type": "object",
        "properties": {
          "model": {
            "type": "string"
          },
          "cpu": {
            "type": "string"
          },
          "ram": {
            "type": "string"
          },
          "hard_drive": {
            "type": "string"
          },
          "year": {
            "type": "integer"
          },
          "screen_size": {
            "type": "string"
          },
          "dollars": {
            "type": "integer"
          },
          "location": {
            "type": "string"
          },
          "condition": {
            "type": "string",
            "enum": [
              "new",
              "like new",
              "excellent",
              "good",
              "fair",
              "salvage"
            ]
          }
        },
        "required": [
          "model",
          "cpu",
          "ram",
          "hard_drive",
          "year",
          "screen_size",
          "dollars",
          "location",
          "condition"
        ],
        "additionalProperties": false
      }
    }
  ]
}

{
  "prompt": "Find the computer specs",
  "temperature": 0,
  "messages": [
    {
      "role": "user",
      "content": "..."
    }
  ],
  "tools": [
    {
      "name": "dataExtractor",
      "description": "Extract data that conforms to the provided schema",
      "parameters": {
        "type": "object",
        "properties": {
          "model": {
            "type": "string"
          },
          "cpu": {
            "type": "string"
          },
          "ram": {
            "type": "string"
          },
          "hard_drive": {
            "type": "string"
          },
          "year": {
            "type": "integer"
          },
          "screen_size": {
            "type": "string"
          },
          "dollars": {
            "type": "integer"
          },
          "location": {
            "type": "string"
          },
          "condition": {
            "type": "string",
            "enum": [
              "new",
              "like new",
              "excellent",
              "good",
              "fair",
              "salvage"
            ]
          }
        },
        "required": [
          "model",
          "cpu",
          "ram",
          "hard_drive",
          "year",
          "screen_size",
          "dollars",
          "location",
          "condition"
        ],
        "additionalProperties": false
      }
    }
  ]
}

{
  "prompt": "Find the computer specs",
  "temperature": 0,
  "messages": [
    {
      "role": "user",
      "content": "..."
    }
  ],
  "tools": [
    {
      "name": "dataExtractor",
      "description": "Extract data that conforms to the provided schema",
      "parameters": {
        "type": "object",
        "properties": {
          "model": {
            "type": "string"
          },
          "cpu": {
            "type": "string"
          },
          "ram": {
            "type": "string"
          },
          "hard_drive": {
            "type": "string"
          },
          "year": {
            "type": "integer"
          },
          "screen_size": {
            "type": "string"
          },
          "dollars": {
            "type": "integer"
          },
          "location": {
            "type": "string"
          },
          "condition": {
            "type": "string",
            "enum": [
              "new",
              "like new",
              "excellent",
              "good",
              "fair",
              "salvage"
            ]
          }
        },
        "required": [
          "model",
          "cpu",
          "ram",
          "hard_drive",
          "year",
          "screen_size",
          "dollars",
          "location",
          "condition"
        ],
        "additionalProperties": false
      }
    }
  ]
}

{
  "prompt": "Find the computer specs",
  "temperature": 0,
  "messages": [
    {
      "role": "user",
      "content": "..."
    }
  ],
  "tools": [
    {
      "name": "dataExtractor",
      "description": "Extract data that conforms to the provided schema",
      "parameters": {
        "type": "object",
        "properties": {
          "model": {
            "type": "string"
          },
          "cpu": {
            "type": "string"
          },
          "ram": {
            "type": "string"
          },
          "hard_drive": {
            "type": "string"
          },
          "year": {
            "type": "integer"
          },
          "screen_size": {
            "type": "string"
          },
          "dollars": {
            "type": "integer"
          },
          "location": {
            "type": "string"
          },
          "condition": {
            "type": "string",
            "enum": [
              "new",
              "like new",
              "excellent",
              "good",
              "fair",
              "salvage"
            ]
          }
        },
        "required": [
          "model",
          "cpu",
          "ram",
          "hard_drive",
          "year",
          "screen_size",
          "dollars",
          "location",
          "condition"
        ],
        "additionalProperties": false
      }
    }
  ]
}

B

beingmudit [WePlayDOS]•2/13/25, 5:41 AM

CF team, can anyone help with yhe above?

Does anyone have any view on which is the best text genertion model that reliably outputs JSON? I'm

Similar Threads

Similar Threads

Similar Threads