Cloudflare Developers•12mo ago

@andrew_nyr might help me?

// sssssslboom @andrew_nyr might help me?

Idle•1/17/25, 6:10 PM

?pings

SuperHelpflare•1/17/25, 6:10 PM

Please do not ping community members for non-moderation reasons. Doing so will not solve your issue faster and will make people less likely to want to help you.

SimonBeno•1/19/25, 10:35 PM

Hello people! New to Workers AI

what are the limits for bge-large embedding models?

In our app, we need to be able to get embeddings for 1 - 2M tokens of text per minute (peak), are the limits high enough?

Super Morris•1/21/25, 1:28 PM

When will DeepSeek-R1 be available?

SSimonBeno Hello people! New to Workers AI what are the limits for bge-large embedding mod...

Bob•1/21/25, 7:37 PM

as per the docs, it has a limit of 1500 req/min. For 2M token, you will need chunk size of 1.3k tokens (or ~1k words).
ref: https://developers.cloudflare.com/workers-ai/platform/limits/
they also might increase the limit for you

SSuper Morris When will DeepSeek-R1 be available? 😄

Advany•1/21/25, 7:41 PM

I was going to ask the same thing...

SSiddanth Reddy and same issue with : @cf/mistral/mistral-7b-instruct-v0.1 ```InferenceUpstreamE...

Siddanth Reddy•1/22/25, 1:07 PM

anyone found any solution related to this error?

BBob as per the docs, it has a limit of 1500 req/min. For 2M token, you will need chu...

SimonBeno•1/22/25, 5:55 PM

Thanks, I meant more the limit of tokens per minute. I couldn't find that. Is there a TPM limit for bge-large?

SSiddanth Reddy anyone found any solution related to this error?

Raylight•1/23/25, 2:03 PM

No solution, unless you count running npx wrangler dev --remotenpx wrangler dev --remote where I thus far haven't been able to reproduce it.

As for the issue, I tried to get to the bottom of it, and it looks like only Cloudflare can solve it. It's seemingly random and when checking the debug logs (with WRANGLER_LOG_SANITIZE=falseWRANGLER_LOG_SANITIZE=false) I noticed that the error coincides with a 401 unauthorized response from upstream.

Ai._parseErrorAi._parseError is unable to interpret the format of the error response, leading to "unknown: unknown". (Looks like there's a recent commit that would fix this specific part.)

crazyjack12•1/23/25, 3:49 PM

Possible to get DeepSeek-R1?

garff•1/23/25, 4:44 PM

Hi, we are thinking about using Worker AI for inference and I'm trying out the different models in the playground. I'm wondering if the inference speed corresponds to what i shown on the screen? Would each request take that long for it to finish? What are the inference speed? How many tokens can I expect to get back in what amount of time? Currenctly I'm testing llama 3.3 70b instruct but I'm open to other models that handles multilingual cases.

Fflashblaze I'm surprised CF has still not addressed this issue. No reply on the community p...

flashblaze•1/24/25, 7:49 AM

I'm appalled at the lack of communication from CF
Still nothing from them

rob•1/24/25, 2:26 PM

the PM for workers AI is on it .. I'm sure

akazwz•1/26/25, 9:30 AM

will Workers AI add more model support function call ？

akazwz•1/26/25, 9:30 AM

for now， just one model support that

Aakazwz will Workers AI add more model support function call ？

ANBU•1/26/25, 8:16 PM

Official only one model supports function call
But I’ve had good experience with llama models as well (I’m using ai gateway tho)
Only limitations is that i wasn’t able to make it do parallel function calling yet

scotto•1/26/25, 8:51 PM

how to ask for limit increase? like RPM

Bob•1/27/25, 11:02 AM

Hello, I am using the "bge-large-en-v1.5" model from cloudflare and same model running in local that I got from huggingface via SentenceTransformer.
For same string I am getting different vectors between cloudflare hosted model and local (screenshots - cloudflare ai in postman vs local with https://huggingface.co/BAAI/bge-large-en-v1.5)

Shouldn't these be exactly same?
One possible reason can be that cloudflare is using a older or different version than the hugging face one.
Also, what is the strategy from cloudflare to support these models? If I use the model for my current data set, I should have access to the model for a very long time to query too.

Bob•1/27/25, 11:40 AM

cosine similarity score between them is [0.9773116111755371]. so they are not very very close

BBob Hello, I am using the "bge-large-en-v1.5" model from cloudflare and same model r...

Isaac McFadyen•1/27/25, 12:04 PM

CUDA is not deterministic meaning you won't get the same outputs every time, especially across GPU models.

Bob•1/27/25, 12:07 PM

oh damn. I didnt know that. thanks for the reply

BBob oh damn. I didnt know that. thanks for the reply

Isaac McFadyen•1/27/25, 12:15 PM

You also aren't passing normalize_embeddings=Truenormalize_embeddings=True locally which the docs recommend (https://huggingface.co/BAAI/bge-large-en-v1.5#using-sentence-transformers).

BAAI/bge-large-en-v1.5 · Hugging Face

Isaac McFadyen•1/27/25, 12:15 PM

I expect if you pass that flag to encode()encode() you might get closer outputs.

Bob•1/27/25, 12:26 PM

still same results. I think cloudflare runs the the models in different architecture or via different libraries than what I am doing.
Chatgpt says pytorch has some different settings for deterministic algo on different GPUs which might be different depending on implementation.

It is not a big deal, I just wanted to have a backup in case cloudflare discontinue this model in future and I might still need to query it for sometime while I re calculate the embeddings with newer model

Bob•1/27/25, 12:38 PM

I ran same query on huggingface now via their api.
it is exact same result that I get in local.
Then I guess cloudflare's architecture/libraries are different

scotto•1/27/25, 12:49 PM

It would be good to know cloud flare preprocess and tokenization process, result should be the same

Bob•1/27/25, 12:59 PM

yes some details around their setup in docs will be helpful for noobs like me

goris29•1/27/25, 5:06 PM

Is there an outage at the moment? I'm getting this returned from my workers AI binding:

{
  "httpCode": 500,
  "message": "Error: Network connection lost.",
  "name": "AiError"
}

{
  "httpCode": 500,
  "message": "Error: Network connection lost.",
  "name": "AiError"
}

I'm seeing this both in the response in my application and in AI Gateway.

It was working fine on Friday when I last tested it.

I'm using the @cf/llava-hf/llava-1.5-7b-hf@cf/llava-hf/llava-1.5-7b-hf model.

Ggotfredsen Can we get DeepSeek R1 on cloudflare Workers AI today ? 🙂 https://github.com/d...

�

😈 Donkey 💫•1/27/25, 6:23 PM

CF AI seems slept for a while, with many many ancient AI models

rob•1/28/25, 12:26 PM

what happened

Ian_X12•1/28/25, 3:38 PM

Is this product dead? Not seeing any recent models.

James•1/28/25, 7:32 PM

Looks like @cf/deepseek-ai/deepseek-r1-distill-qwen-32b@cf/deepseek-ai/deepseek-r1-distill-qwen-32b just landed in Workers AI

Advany•1/29/25, 10:41 AM

did you add it to toml? [ai]
binding = "AI"

topmass•1/29/25, 5:33 PM

Can someone explain the pricing for deepseek-r1-distill-qwen-32b? - most distillations below 70b are a bit underwhelming on performance and the 32b version is priced higher than the full R1 on deepseek api ?

Ttopmass Can someone explain the pricing for deepseek-r1-distill-qwen-32b? - most distill...

MattIPv4•1/29/25, 7:03 PM

Cf: https://developers.cloudflare.com/workers-ai/models/deepseek-r1-distill-qwen-32b/

$0.50 per 1M input token
$4.88 per 1M output token

DeepSeek: https://api-docs.deepseek.com/quick_start/pricing (deepseek-reasoner)

$0.55 per 1M input token (cache miss; $0.14 if cache hit)
$2.19 per 1M output token

Yeh not sure how to explain that one unfortunately

BBob Hello, I am using the "bge-large-en-v1.5" model from cloudflare and same model r...

Bob•1/29/25, 7:08 PM

finally found the reason why cloudflare ai and hugging face endpoint/my local give different results

hugging face endpoint most likely using "sentence_transformers" and cloudflare ai using "transformers" with normalization.
now I can die in peace

MMattIPv4 Cf: <https://developers.cloudflare.com/workers-ai/models/deepseek-r1-distill-qwe...

topmass•1/29/25, 7:32 PM

This is also comparing the Full (non distilled) R1 to 32b distilled R1 reasoning Qwen model. In extensive testing over the past few days I could not confidently recommend anything below 70b distilled variant for production, the thinking tokens are decent on all but when acting upon them the performance is still a bit hodge-podge. You can see this in use cases like asking for simple structured outputs or json function calling - I hope workers ai will consider lowering the price significantly or offering the 70b distilled variant if wanting to charge this high

rob•1/29/25, 10:30 PM

yeaaaaa lets go AGENTS~

rob•1/30/25, 2:18 PM

we want full r1!!! not that "we have r1 at home" version plz

@andrew_nyr might help me?

Similar Threads

Similar Threads

Similar Threads