hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does not return promptTo

hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does not return promptToken ? it just return zero

GitHub

GitHub - cloudflare/workers-ai-provider: A Workers AI provider for ...

A Workers AI provider for the vercel AI SDK. Contribute to cloudflare/workers-ai-provider development by creating an account on GitHub.

BumblebeeSquareOP•2/10/25, 1:22 AM

cant use this with ai sdk at all

or am i missig something, is there a way to get usage out of it

AAndrew hi does anyone know if ai-gateway supports streaming response for Google AI Stu...

usualdev•2/10/25, 1:44 PM

I'm using it from OpenRouter and it works fine.

Hurricane•2/10/25, 3:58 PM

Hey guys, can anyone tell me how to download the AI Gateway logs in order to create an AI dataset for LLM fine tuning?
I already created a dataset from a date range, but I can't do anything with it besides deleting it.

HHurricane Hey guys, can anyone tell me how to download the AI Gateway logs in order to cre...

Hurricane•2/10/25, 10:51 PM

I found a quite cumbersome way but better than nothing:

List Gateway Logs:
curl https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways/$GATEWAY_ID/logs \
-H "X-Auth-Email: $CLOUDFLARE_EMAIL" \
-H "X-Auth-Key: $CLOUDFLARE_API_KEY"

Then, for every found ID:
curl https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways/$GATEWAY_ID/logs/$ID \
-H "X-Auth-Email: $CLOUDFLARE_EMAIL" \
-H "X-Auth-Key: $CLOUDFLARE_API_KEY"

rob•2/11/25, 3:48 PM

^^ thank u

HHurricane Hey guys, can anyone tell me how to download the AI Gateway logs in order to cre...

Kathy•2/12/25, 10:25 PM

https://developers.cloudflare.com/ai-gateway/observability/logging/logpush/

Cloudflare Docs

Workers Logpush · Cloudflare AI Gateway docs

AI Gateway allows you to securely export logs to an external storage location, where you can decrypt and process them.
You can toggle Workers Logpush on and off in the Cloudflare dashboard settings. This product is available on the Workers Paid plan. For pricing information, refer to Pricing.

Puliczek•2/13/25, 8:50 AM

Hi, I'm experiencing an issue with the CF AI Gateway while using Google Gemini Flash 2.0.

About 90% of the time, I receive the following error:

423: 'Resource has been exhausted (e.g., check quota).'

Here are some details about my situation:

My API key is on a paid plan with Gemini.
I send a relatively low number of requests: 1-5 per minute, totaling around 1,000 per month.
When I bypass the gateway and use a direct curl request from my PC, I don't encounter any blocking issues.
Could you help me resolve this?

Puliczek•2/13/25, 8:53 AM

Puliczek•2/13/25, 8:54 AM

at 8:32:14 AM there was only one API call, and it got also 429.

Puliczek•2/13/25, 10:53 AM

No, I turned off gateway, works locally, but deployed to cloudflare pages has same error.

Puliczek•2/13/25, 10:53 AM

Puliczek•2/13/25, 10:54 AM

i think i will just switch to openai or llama model.

Puliczek•2/13/25, 3:32 PM

changed to openai, and now it works great.

Lila Rest•2/13/25, 8:50 PM

Hey similar error here, our AI requests are failing with the following rate limit error, while the volume is relatively low ($20 of consumption). That heavily impacts our services
{ message: 'AiError: AiError: Capacity temporarily exceeded, please try again.', code: 3040 } { message: 'AiError: AiError: Capacity temporarily exceeded, please try again.', code: 3040 }

LLila Rest Hey similar error here, our AI requests are failing with the following rate limi...

Kathy•2/14/25, 12:24 AM

is this with google too?

KKathy is this with google too?

Isaac McFadyen•2/14/25, 4:51 AM

Given their post in workers-ai I'm guessing it was a seperate issue with Workers AI.

IIsaac McFadyen Given their post in https://discord.com/channels/595317990191398933/110547700996...

Lila Rest•2/14/25, 9:23 AM

Indeed, I had posted in the wrong channel!

thomash•2/17/25, 12:55 AM

Is it impossible to supply a custom openai endpoint url to AI gateway? We are using models from providers such as scaleway which are openai compatible but require a custom endpoint url. we also have some models our community finetuned which are deployed in different openai compatible services like modal.com.

just searched this channel a little and it seems the answer is no. i guess not using AI gateway but hosting a litellm server somewhere is the most practical solution

thomash•2/17/25, 5:01 AM

LiteLLM doesn't seem to be production ready

If anyone else is stuck with this I found portkey a very good alternative:

https://github.com/Portkey-AI/gateway

GitHub

GitHub - Portkey-AI/gateway: A blazing fast AI Gateway with integra...

A blazing fast AI Gateway with integrated guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. - Portkey-AI/gateway

R3AP3R editz•2/20/25, 1:53 PM

ummm
is there extra chharge?

RR3AP3R editz ummm is there extra chharge?

Unsmart•2/20/25, 7:10 PM

You wont get charged for going over the limit. Sometimes limits dont kick in right away so you just get extra free usage

R3AP3R editz•2/20/25, 7:21 PM

Ohh ok but another issue is in that neurons
4.6k was of yesterday
And I was waiting for it to reset but it didn't and continued

R3AP3R editz•2/20/25, 7:21 PM

Why is that?

Unsmart•2/20/25, 7:22 PM

Yesterday in your timezone might not be yesterday in UTC. All limits go by UTC time (7:22PM now).

UUnsmart Yesterday in your timezone might not be yesterday in UTC. All limits go by UTC t...

R3AP3R editz•2/20/25, 7:23 PM

Ik in mine is IST and 12am UTC is 5 am IST but it didn't change

R3AP3R editz•2/20/25, 7:23 PM

And this is not first time

R3AP3R editz•2/20/25, 7:23 PM

For me

R3AP3R editz•2/20/25, 7:32 PM

Another one is in model:@hf/nousresearch/hermes-2-pro-mistral-7b

Its says that this model supports function falling
In https://developers.cloudflare.com/workers-ai/models/hermes-2-pro-mistral-7b/

But I am getting how to make it to connect to external APIs like for example web search mainly
I tried but it replied as it doesnt have function calling capability
So anyone please explain me

Cloudflare Docs

hermes-2-pro-mistral-7b · Cloudflare Workers AI docs

Hermes 2 Pro on Mistral 7B is the new flagship 7B Hermes! Hermes 2 Pro is an upgraded, retrained version of Nous Hermes 2, consisting of an updated and cleaned version of the OpenHermes 2.5 Dataset, as well as a newly introduced Function Calling and JSON Mode dataset developed in-house.

RR3AP3R editz Another one is in model:@hf/nousresearch/hermes-2-pro-mistral-7b Its says that ...

Victor•2/20/25, 8:36 PM

@hf/nousresearch/hermes-2-pro-mistral-7b@hf/nousresearch/hermes-2-pro-mistral-7b is the only one makred as Function callingFunction calling in the docs/api, but all the llama 3s (~~3.0,~~ 3.1, 3.2, and 3.3) advertise function calling in their respective blog posts. In my personal testing, hermeshermes fails like 70% of the time, but llama 3.2 & 3.3 worked > 95% of the time

VVictor `@hf/nousresearch/hermes-2-pro-mistral-7b` is the only one makred as `Function c...

R3AP3R editz•2/21/25, 12:42 AM

Wait those support function calling?
Means it can also access external api?
Hey could u please say how to add it
Because i want to add api for web search so

UUnsmart Yesterday in your timezone might not be yesterday in UTC. All limits go by UTC t...

R3AP3R editz•2/21/25, 2:04 AM

See its cross 0:00 UTC BUT IT ONLY GOT DESCRESED BY 3.87K

Screenshot_2025-02-21-06-43-36-70_40deb401b9ffe8e1df2f1cc5ba480b12.jpg

fatwang2•2/24/25, 1:26 AM

hi
Team, will you think about support together and fireworks or even some more OpenAI-Compatible Endpoints?

fatwang2•2/24/25, 1:27 AM

I am using AI gateway on my service now, but it only support some of my providers

Ffatwang2 hi Team, will you think about support together and fireworks or even some more O...

Kathy•2/25/25, 8:47 PM

we take requests

so will take into account your request for together + fireworks as we prioritize

Ffatwang2 hi Team, will you think about support together and fireworks or even some more O...

Kathy•2/25/25, 8:48 PM

when you say "some more OpenAI-Compatible Endpoints" what do you mean? Do you mean adding more ai providers, like together and fireworks that are openai compatible? or something else

KKathy when you say "some more OpenAI-Compatible Endpoints" what do you mean? Do you me...

fatwang2•2/25/25, 9:07 PM

Too many providers for you to add,but most of them are OpenAI-Compatible-Endpoints, I am thinking if you can add a feature to let us input the API base that we want ai gateway request.

just like litellm or porkey and other ai gateway products do.

Bryan Chen•2/26/25, 2:09 AM

for custom costs, does it able to take cache hit/miss into consideration? otherwise is it possible for me to calculate a final cost and pass it? for example, I can get the request cost from OpenRouter directly and would like to have it recorded

tkisme•2/26/25, 4:18 AM

Is output token limit to 256 now?

Ttkisme Is output token limit to 256 now?

James•2/26/25, 2:54 PM

defaults are enforced now it seems, which hopefully will be more loudly communicated soon, as it can cause breaking changes for people's apps. cc @michelle
You should be able to increase max_tokensmax_tokens to tweak this.

Johnny Bernhard•2/27/25, 3:10 PM

Is there a way to limit the new guardrail to specific operations in a gateway?
For our use-case it would be enough to enable the guardrails on the embedding step before RAG. We don't need to run them afterwards.

Ffatwang2 Too many providers for you to add,but most of them are OpenAI-Compatible-Endpoin...

thomash•2/28/25, 4:53 PM

i ended up using the portkey ai gateway.

https://github.com/Portkey-AI/gateway

the good thing is you can deploy it straight to cloudflare workers and it allows using custom openai endpoints

GitHub

GitHub - Portkey-AI/gateway: A blazing fast AI Gateway with integra...

A blazing fast AI Gateway with integrated guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. - Portkey-AI/gateway

Tthomash i ended up using the portkey ai gateway. https://github.com/Portkey-AI/gateway...

fatwang2•2/28/25, 7:09 PM

I am on litellm now..

fatwang2•2/28/25, 7:09 PM

similar to portkey

Ffatwang2 I am on litellm now..

thomash•2/28/25, 7:33 PM

i looked at both and portkey seemed more production ready and easy to deploy serverless to cloudflare. but im sure litellm is also fine. it has many users

Tthomash i looked at both and portkey seemed more production ready and easy to deploy ser...

fatwang2•3/1/25, 6:13 AM

how about the latency

Ffatwang2 how about the latency

thomash•3/1/25, 6:44 PM

portkey is much faster but usually latency is constrained by the LLM not the gateway

numberstorm•3/2/25, 9:47 AM

hey, how do i give access to my domain to a friend

JJohnny Bernhard Is there a way to limit the new guardrail to specific operations in a gateway? F...

Kathy•3/3/25, 5:37 PM

hm so you would not want to run on the prompts where you're querying RAG? (this "knowledge query" part of RAG https://developers.cloudflare.com/reference-architecture/diagrams/ai/ai-rag/#knowledge-queries

For embeddings, we only evaluate the prompts https://developers.cloudflare.com/ai-gateway/guardrails/supported-model-types/

Cloudflare Docs

Supported model types · Cloudflare AI Gateway docs

AI Gateway's Guardrails detects the type of AI model being used and applies safety checks accordingly:

Cloudflare Docs

Retrieval Augmented Generation (RAG) · Cloudflare Reference Archite...

Retrieval-Augmented Generation (RAG) is an innovative approach in natural language processing that integrates retrieval mechanisms with generative models to enhance text generation.

BBryan Chen for custom costs, does it able to take cache hit/miss into consideration? otherw...

Kathy•3/3/25, 6:37 PM

if it's a cache HIT the cost will be 0. Even if you have passed a custom cost

hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does not return promptTo

Similar Threads

Similar Threads

Similar Threads