are you using google vertex or google ai studio? i just tested on google ai studio and seems like ca

are you using google vertex or google ai studio? i just tested on google ai studio and seems like caching is working properly. Have you set any different cache configuration for gemini? https://developers.cloudflare.com/ai-gateway/configuration/caching/

Cloudflare Docs

Caching · Cloudflare AI Gateway docs

Override caching settings on a per-request basis.

BBumblebeeSquare btw the cost for gemini flash looks diffferent from my calculation: 415 * 0.07...

KathyOP•1/29/25, 12:17 AM

thanks for pointing out. ill get that fixed

KKathy we do take into account requests! thanks for letting me know. curious why you'...

laurynas•1/29/25, 6:23 AM

thank you! Experimenting still but mostly due to speed. LLaMa on Cerebras architecture outputs at 2k tokens/sec which makes a world of difference in terms of latency in the UX

BumblebeeSquare•1/29/25, 6:58 PM

Hi guys, is there a way to do analytic, collect token costs without enable logs ? i feel my user prob dont want me to see their inappropriate prompts in logs

KKathy are you using google vertex or google ai studio? i just tested on google ai stud...

tornado•1/29/25, 7:03 PM

I'm using Google AI studio and no changes to default caching. I did more investigation and it seems like when I send request locally things show up on the dashboard but inside my deployed cloudflare workflow only the Claude api calls show up in the dashboard.

rob•1/30/25, 11:24 PM

https://x.com/ritakozlov_/status/1885034425538187610

rita kozlov 🐀 (@ritakozlov_) on X

we shipped a binding you can use for calling @cloudflaredev ai gateway directly from a worker! if you're already using workers + workers ai, just add this to your existing code:

gateway: {
id: "my-gateway"
}
or send granular feedback w your logs →
https://t.co/ZZIvA04kPb

•

1/30/25, 6:36 PM

Rrob https://x.com/ritakozlov_/status/1885034425538187610

KathyOP•1/30/25, 11:35 PM

hope yall like!

Zig•1/31/25, 12:42 AM

Does anyone know how to use the websockets api with streaming? Im trying to use the universal endpoint with openai.

If i send a content type application/json header with no streaming it works fine, if i try and do it with streaming i see the request going through on the dashboard but my worker never receives any websocket messages. if i send it without a content type header or with event-stream i allways get an error about a missing model

KKathy what model are you using from anthropic? i can check

dom•2/3/25, 2:06 PM

Using claude-3-5-sonnet-latest which currently defaults to claude-3-5-sonnet-20241022

I see o3 mini also has this issue.

usualdev•2/5/25, 9:02 PM

Feature request, add ability to adjust the column length. I don't want feedback, latency, or status to be that wide, I would rather see the model name

VIO•2/6/25, 2:18 PM

url not working https://developers.cloudflare.com/ai-gateway/request-handling/

VVIO url not working https://developers.cloudflare.com/ai-gateway/request-handling/

KathyOP•2/6/25, 7:46 PM

how did you get to this url?
correct one is: https://developers.cloudflare.com/ai-gateway/configuration/request-handling/

Cloudflare Docs

Request handling · Cloudflare AI Gateway docs

Your AI gateway supports different strategies for handling requests to providers, which allows you to manage AI interactions effectively and ensure your applications remain responsive and reliable.

KathyOP•2/6/25, 7:47 PM

nevermind i found it- the changelog THANKS

rob•2/7/25, 2:08 AM

https://x.com/CloudflareDev/status/1887636157963649046

Cloudflare Developers (@CloudflareDev) on X

We've got 2 new exciting updates for AI Gateway!

•Gain observability and control over your Cerebras,
ElevenLabs and Cartesia usage via AI Gateway.

•Add more control to your requests with new timeout, retry, and fallback options.

Learn more about these two updates below

•

2/6/25, 10:55 PM

luiseok•2/7/25, 3:23 AM

Does anyone get errors when trying to google gemini 1.5 flash? I know it's not an ai gateway problem, but just to be sure if I'm the only one having trouble.

rob•2/7/25, 2:25 PM

Kavatch•2/7/25, 4:55 PM

There is no universal way to use ai gateway with any openai compatabil endpoint correct?
I am using deepinfra and would like to use ai gateway however I don't see any options to set my own base url

morpig•2/8/25, 11:13 AM

hi! wondering if there any plans to return request ID (what we see in AI gateway logs) in workers. or is it possible already?

GGabriel const myLogId = env.AI.aiGatewayLogId;

morpig•2/9/25, 12:34 PM

my bad for not looking further! this is awesome. thank you!

BumblebeeSquare•2/10/25, 1:21 AM

hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does not return promptToken ? it just return zero

GitHub

GitHub - cloudflare/workers-ai-provider: A Workers AI provider for ...

A Workers AI provider for the vercel AI SDK. Contribute to cloudflare/workers-ai-provider development by creating an account on GitHub.

BumblebeeSquare•2/10/25, 1:22 AM

cant use this with ai sdk at all

or am i missig something, is there a way to get usage out of it

AAndrew hi does anyone know if ai-gateway supports streaming response for Google AI Stu...

usualdev•2/10/25, 1:44 PM

I'm using it from OpenRouter and it works fine.

Hurricane•2/10/25, 3:58 PM

Hey guys, can anyone tell me how to download the AI Gateway logs in order to create an AI dataset for LLM fine tuning?
I already created a dataset from a date range, but I can't do anything with it besides deleting it.

HHurricane Hey guys, can anyone tell me how to download the AI Gateway logs in order to cre...

Hurricane•2/10/25, 10:51 PM

I found a quite cumbersome way but better than nothing:

List Gateway Logs:
curl https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways/$GATEWAY_ID/logs \
-H "X-Auth-Email: $CLOUDFLARE_EMAIL" \
-H "X-Auth-Key: $CLOUDFLARE_API_KEY"

Then, for every found ID:
curl https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways/$GATEWAY_ID/logs/$ID \
-H "X-Auth-Email: $CLOUDFLARE_EMAIL" \
-H "X-Auth-Key: $CLOUDFLARE_API_KEY"

rob•2/11/25, 3:48 PM

^^ thank u

HHurricane Hey guys, can anyone tell me how to download the AI Gateway logs in order to cre...

KathyOP•2/12/25, 10:25 PM

https://developers.cloudflare.com/ai-gateway/observability/logging/logpush/

Cloudflare Docs

Workers Logpush · Cloudflare AI Gateway docs

AI Gateway allows you to securely export logs to an external storage location, where you can decrypt and process them.
You can toggle Workers Logpush on and off in the Cloudflare dashboard settings. This product is available on the Workers Paid plan. For pricing information, refer to Pricing.

Puliczek•2/13/25, 8:50 AM

Hi, I'm experiencing an issue with the CF AI Gateway while using Google Gemini Flash 2.0.

About 90% of the time, I receive the following error:

423: 'Resource has been exhausted (e.g., check quota).'

Here are some details about my situation:

My API key is on a paid plan with Gemini.
I send a relatively low number of requests: 1-5 per minute, totaling around 1,000 per month.
When I bypass the gateway and use a direct curl request from my PC, I don't encounter any blocking issues.
Could you help me resolve this?

Puliczek•2/13/25, 8:53 AM

Puliczek•2/13/25, 8:54 AM

at 8:32:14 AM there was only one API call, and it got also 429.

Puliczek•2/13/25, 10:53 AM

No, I turned off gateway, works locally, but deployed to cloudflare pages has same error.

Puliczek•2/13/25, 10:53 AM

Puliczek•2/13/25, 10:54 AM

i think i will just switch to openai or llama model.

Puliczek•2/13/25, 3:32 PM

changed to openai, and now it works great.

Lila Rest•2/13/25, 8:50 PM

Hey similar error here, our AI requests are failing with the following rate limit error, while the volume is relatively low ($20 of consumption). That heavily impacts our services

 {       message: 'AiError: AiError: Capacity temporarily exceeded, please try again.',       code: 3040     }

 {       message: 'AiError: AiError: Capacity temporarily exceeded, please try again.',       code: 3040     }

LLila Rest Hey similar error here, our AI requests are failing with the following rate limi...

KathyOP•2/14/25, 12:24 AM

is this with google too?

KKathy is this with google too?

Isaac McFadyen•2/14/25, 4:51 AM

Given their post in workers-ai I'm guessing it was a seperate issue with Workers AI.

IIsaac McFadyen Given their post in https://discord.com/channels/595317990191398933/110547700996...

Lila Rest•2/14/25, 9:23 AM

Indeed, I had posted in the wrong channel!

thomash•2/17/25, 12:55 AM

Is it impossible to supply a custom openai endpoint url to AI gateway? We are using models from providers such as scaleway which are openai compatible but require a custom endpoint url. we also have some models our community finetuned which are deployed in different openai compatible services like modal.com.

just searched this channel a little and it seems the answer is no. i guess not using AI gateway but hosting a litellm server somewhere is the most practical solution

thomash•2/17/25, 5:01 AM

LiteLLM doesn't seem to be production ready

If anyone else is stuck with this I found portkey a very good alternative:

https://github.com/Portkey-AI/gateway

GitHub

GitHub - Portkey-AI/gateway: A blazing fast AI Gateway with integra...

A blazing fast AI Gateway with integrated guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. - Portkey-AI/gateway

R3AP3R editz•2/20/25, 1:53 PM

ummm
is there extra chharge?

RR3AP3R editz ummm is there extra chharge?

Unsmart•2/20/25, 7:10 PM

You wont get charged for going over the limit. Sometimes limits dont kick in right away so you just get extra free usage

R3AP3R editz•2/20/25, 7:21 PM

Ohh ok but another issue is in that neurons
4.6k was of yesterday
And I was waiting for it to reset but it didn't and continued

R3AP3R editz•2/20/25, 7:21 PM

Why is that?

Unsmart•2/20/25, 7:22 PM

Yesterday in your timezone might not be yesterday in UTC. All limits go by UTC time (7:22PM now).

UUnsmart Yesterday in your timezone might not be yesterday in UTC. All limits go by UTC t...

R3AP3R editz•2/20/25, 7:23 PM

Ik in mine is IST and 12am UTC is 5 am IST but it didn't change

R3AP3R editz•2/20/25, 7:23 PM

And this is not first time

R3AP3R editz•2/20/25, 7:23 PM

For me

R3AP3R editz•2/20/25, 7:32 PM

Another one is in model:@hf/nousresearch/hermes-2-pro-mistral-7b

Its says that this model supports function falling
In https://developers.cloudflare.com/workers-ai/models/hermes-2-pro-mistral-7b/

But I am getting how to make it to connect to external APIs like for example web search mainly
I tried but it replied as it doesnt have function calling capability
So anyone please explain me

Cloudflare Docs

hermes-2-pro-mistral-7b · Cloudflare Workers AI docs

Hermes 2 Pro on Mistral 7B is the new flagship 7B Hermes! Hermes 2 Pro is an upgraded, retrained version of Nous Hermes 2, consisting of an updated and cleaned version of the OpenHermes 2.5 Dataset, as well as a newly introduced Function Calling and JSON Mode dataset developed in-house.

are you using google vertex or google ai studio? i just tested on google ai studio and seems like ca

Similar Threads