Cloudflare Developers•11mo ago

Two main criteria: - The bigger the weight the more intelligent they will (generally) be, i.e. a 70b

Two main criteria:

The bigger the weight the more intelligent they will (generally) be, i.e. a 70b will usually beat an 8b although newer 8b might come close, but also the slower they will return a response
The provider who trained it, i.e. Mistral or Llama are both good options but it's really just a matter of testing to see which is better for your specific use-case

IIsaac McFadyen Two main criteria: - The bigger the weight the more intelligent they will (gener...

Mitya•2/4/25, 7:00 PM

That's helpful - thanks!

Marius B.•2/5/25, 9:23 AM

Someone please help me understand Workers AI vs Deepseek Pricing. The DeepSeek API costs $0.28 per million output tokens. Running DeepSeek on Workers would cost $4.88 per 1M output token + the unbound worker usage, I assume. But this is a big discrepancy in price. Am I getting something wrong?
https://api-docs.deepseek.com/quick_start/pricing
https://developers.cloudflare.com/workers-ai/models/deepseek-r1-distill-qwen-32b/

MMattIPv4 Cf: <https://developers.cloudflare.com/workers-ai/models/deepseek-r1-distill-qwe...

Advany•2/5/25, 10:24 AM

already anwsered, scroll back

rob•2/5/25, 3:29 PM

tldr. CHOYNA

Mitya•2/5/25, 3:46 PM

Does anyone have any view on which is the best text genertion model that reliably outputs JSON? I'm finding that llama-3.3-70b-instruct-fp8-fastllama-3.3-70b-instruct-fp8-fast is reliable about 70% of the time, but will occasionally throw in an invalid bracket here and there. Any other models people are using for this? (Yes I know there's Function Calls as well.)

MMitya Does anyone have any view on which is the best text genertion model that reliabl...

Isaac McFadyenOP•2/5/25, 6:28 PM

Your best bet is to implement retries, really.

Isaac McFadyenOP•2/5/25, 6:28 PM

Try parsing the JSON and if it fails re-send the request.

Isaac McFadyenOP•2/5/25, 6:28 PM

Unless specifically forced to output JSON via low-level logit math (which Workers AI doesn't support, I don't think) then there will always be some cases where it returns invalid JSON.

Isaac McFadyenOP•2/5/25, 6:29 PM

You can try to improve it by using specific system messages but it'll never be 100% without forced logits.

IIsaac McFadyen Your best bet is to implement retries, really.

Mitya•2/6/25, 9:27 AM

Thanks. Yeah this is what I'm doing currently. It's helping a lot, but as you say, there's always some cases. I note that the mistral model (CF's single function calls-capable model, at least according to the filters on the models page) has a "JSON mode". But in the input params, this is not mentioned, so supposedly there's a way to force it to return JSON but I'm unsure whether there's a param missing from the docs.

Mitya•2/6/25, 9:28 AM

Also, this looks interesting re: structured output: https://js.langchain.com/docs/how_to/structured_output/

How to return structured data from a model | 🦜️🔗 Langchain

It is often useful to have a model return output that matches some

MMitya Thanks. Yeah this is what I'm doing currently. It's helping a lot, but as you sa...

Isaac McFadyenOP•2/6/25, 1:01 PM

I don't believe there is. That's for when you're running it locally.

MMitya Thanks. Yeah this is what I'm doing currently. It's helping a lot, but as you sa...

thomash•2/6/25, 2:04 PM

if you are using the OpenAI compatible API, I think you can enable JSON mode by passing the correct response_formatresponse_format. Since it is possible to access of CloudFlare's models via the OpenAI API.

Tthomash if you are using the OpenAI compatible API, I think you can enable JSON mode by ...

Isaac McFadyenOP•2/6/25, 2:15 PM

Cloudflare's API does not respect the JSON mode setting iirc

Isaac McFadyenOP•2/6/25, 2:15 PM

I tried the other day and it ignored it.

Torianna•2/6/25, 2:16 PM

to be fair, json mode is only supported on a small amount of models

Torianna•2/6/25, 2:16 PM

since it requires modifying how the model responds

TTorianna to be fair, json mode is only supported on a small amount of models

Isaac McFadyenOP•2/6/25, 2:17 PM

Kind of? Like, the models themselves have "JSON mode" but it's not actually that hard to force it to be JSON, it just needs some forced logits but theoretically any backend that does that will work with any model.

Isaac McFadyenOP•2/6/25, 2:17 PM

I just don't think CF's backend has that setting or if it does it's not enabled.

Torianna•2/6/25, 2:18 PM

true

Torianna•2/6/25, 2:18 PM

but also looking at the models CF has, most dont allow modifying the logit bias

Muhammadreza Haghiri•2/6/25, 2:24 PM

Hi everyone.
I am using CF AI workers a lot and recently, my startup (basically me and my team) relesead this new 8B multilingual model:

https://huggingface.co/mann-e/Hormoz-8B

I just want to know how can we use it through AI workers?

mann-e/Hormoz-8B · Hugging Face

MMuhammadreza Haghiri Hi everyone. I am using CF AI workers a lot and recently, my startup (basically...

Isaac McFadyenOP•2/6/25, 2:32 PM

?workers-ai-models

SuperHelpflare•2/6/25, 2:32 PM

Workers AI currently only supports popular open-source models provided by the Cloudflare team, as well as your own LoRAs that can be applied on top of the Cloudflare-provided models. You cannot currently upload your own models or use a model from HuggingFace. See the documentation for the list of Cloudflare-provided models: https://developers.cloudflare.com/workers-ai/models/

Mitya•2/6/25, 4:29 PM

Update on the structured/JSON output discussion above, I've had great results with a mixture of LangChain, native JSON schemas and Mistral (when using Mistral directly via their own API). Here's a truncated example:

import { ChatMistralAI } from "@langchain/mistralai"
router.get('/mistral', async req => {
  const model = new ChatMistralAI({
    model: "mistral-large-latest",
    temperature: 0,
    apiKey: <MISTRAL-API-KEY>
  });
  const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>);
  return await structuredLlm.invoke(<YOUR-PROMPT-HERE>);
})

import { ChatMistralAI } from "@langchain/mistralai"
router.get('/mistral', async req => {
  const model = new ChatMistralAI({
    model: "mistral-large-latest",
    temperature: 0,
    apiKey: <MISTRAL-API-KEY>
  });
  const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>);
  return await structuredLlm.invoke(<YOUR-PROMPT-HERE>);
})

Returns JSON nearly every time and there's support for built-in retries where it fails.

crazyjack12•2/6/25, 7:11 PM

There was a beta to be able to use a hugging face model, but not sure what’s the state of that project @Muhammadreza Haghiri

dandoen•2/7/25, 11:44 AM

maybe this channel is more appropriate than vectorize, but i'm wondering if there are any short term plans to make the multi-lingual text embedding model BAAI/bge-m3 available? current BGE ones are EN only
vectorize

GaryLi•2/7/25, 4:11 PM

asking if workers ai can get full-sized deepseek r1 model

.triloksingh•2/7/25, 4:48 PM

#workers-aiI have a paid plan, but I want to make expectations for my admin panel. my website url is example.com/admin

prisum 🎄•2/8/25, 1:01 AM

so if I upgrade to the $5 a month plan I get infinite ai requests?

Pprisum 🎄so if I upgrade to the $5 a month plan I get infinite ai requests?

GaryLi•2/8/25, 10:42 AM

go to docs, theres s pricing list for ai

Puliczek•2/8/25, 12:36 PM

what type of MCP is used by worker-mcp? command or SSE?

nukely•2/8/25, 5:37 PM

any tutorial on how to use Gemini Multimodal Live API with nextjs and a websocket server

GGaryLi go to docs, theres s pricing list for ai

prisum 🎄•2/8/25, 6:49 PM

Yeah I did but on the direct worker ai page it says 30,000 requests monthly but on the pricing document it says unlimited

Pprisum 🎄Yeah I did but on the direct worker ai page it says 30,000 requests monthly but ...

Nutjob 홍•2/8/25, 10:18 PM

Which page are you looking at?

https://developers.cloudflare.com/workers-ai/platform/pricing/

10,000 free tokens a day and then pricing based on model size and blended input/output tokens.

Cloudflare Docs

Pricing · Cloudflare Workers AI docs

Workers AI is included in both the Free and Paid Workers plans and is priced based on model task, model size, and units.

Eelement14 I would also like to know if there is any REST API Endpoint (or via wrangler com...

user•2/9/25, 7:59 AM

Anything on this?

Muhammadreza Haghiri•2/9/25, 3:57 PM

Will FLUX-Dev be added to the models as well?

shoo•2/9/25, 5:09 PM

InferenceUpstreamError: undefined: undefined

anyone who knows workaorund to this?

if I use --remote, it doesn't connect to any other worker in local as workers binding

Bob•2/9/25, 7:23 PM

Hello. I have a question.
bge-large-en-v1.5 <- for this model, how many text items I can send in 1 request? I can't seem to find any info on it.

BBumblebeeSquare hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does...

BumblebeeSquare•2/10/25, 9:18 PM

leisure•2/11/25, 6:19 AM

Hi, I have a question. When I use it, I find that Daily Usage Limits will not reset at zero normally. Is this a special problem?

BBumblebeeSquare Empty message

Raylight•2/11/25, 11:27 AM

Yes. It's not implemented in the package. I created a quick fix (https://github.com/Raylight-PWL/workers-ai-provider/pull/1) but haven't used/tested the package. In any case, you can look at the diff if you want to patch it yourself.

GGaryLi asking if workers ai can get full-sized deepseek r1 model

GaryLi•2/11/25, 3:34 PM

idk why they dont put this on

rob•2/11/25, 3:48 PM

i dont think they can handle the size

rob•2/11/25, 3:48 PM

yet

Marco Aleixo•2/11/25, 7:51 PM

do you plan to add more embedding models?
only having a single not multi lingual model available for embedding is kind of a big blocker to use workers ai

beingmudit [WePlayDOS]•2/12/25, 9:08 PM

Hi CF team, had a doubt about function calling and the supported model. I can see that only one model is supported but in the documentation of this llama model - https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct/

I can see in the input/output params, we can pass tools. Is there a documentation mismatch?

Cloudflare Docs

llama-3.2-11b-vision-instruct · Cloudflare Workers AI docs

The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image.

import { ChatMistralAI } from "@langchain/mistralai" router.get('/mistral', async req => { const model = new ChatMistralAI({ model: "mistral-large-latest", temperature: 0, apiKey: <MISTRAL-API-KEY> }); const structuredLlm = model.withStructuredOutput(<JSON-SCHEMA-HERE>); return await structuredLlm.invoke(<YOUR-PROMPT-HERE>); })

Two main criteria: - The bigger the weight the more intelligent they will (generally) be, i.e. a 70b

Similar Threads

Similar Threads

Similar Threads