With OpenRouter, I can use the key `models` to send up to 3 models and OpenRouter will pick one base

With OpenRouter, I can use the key

models

models

models

models to send up to 3 models and OpenRouter will pick one based on availability. So load balancing is already managed by them.

However, I can't see this in the Proxy options. Is that supported yet?

Docs: https://openrouter.ai/docs/requests#request-body

Example:

// See "Model Routing" section: openrouter.ai/docs/model-routing
  models?: string[];

// See "Model Routing" section: openrouter.ai/docs/model-routing
  models?: string[];

// See "Model Routing" section: openrouter.ai/docs/model-routing
  models?: string[];

// See "Model Routing" section: openrouter.ai/docs/model-routing
  models?: string[];

OpenRouter

Requests | OpenRouter

Handle incoming and outgoing requests

Uusualdev With OpenRouter, I can use the key `models` to send up to 3 models and OpenRoute...

Isaac McFadyen•1/15/25, 9:55 PM

That example is using the openaiopenai client and is just an example of how you could use it. I expect that you can use however you normally connect to OpenRouter and it'll work with that endpoint.

IIsaac McFadyen That example is using the `openai` client and is just an example of how you coul...

usualdevOP•1/15/25, 10:02 PM

Thanks Isaac. I have tested it now and it seems to work fine. Would be great to add that to the docs. Also I tested adding custom headers and that worked well too.

Uusualdev Thanks Isaac. I have tested it now and it seems to work fine. Would be great to ...

Kathy•1/16/25, 9:03 PM

docs are opensource! so if you find any bugs/think things should be added please feel free to PR

Bob•1/19/25, 9:04 AM

is it possible to use ai gateway for a provider outside the list of providers in the doc? If somehow we can pass their endpoint and payload directly that would be great.

BBob is it possible to use ai gateway for a provider outside the list of providers in...

Kathy•1/21/25, 6:40 PM

what provider are you using? we take into account user requests/demand when adding to our provider list!

KKathy what provider are you using? we take into account user requests/demand when addi...

Bob•1/21/25, 7:26 PM

Voyage AI for embedding generation

BBob Voyage AI for embedding generation

Kathy•1/21/25, 11:24 PM

noted - this is the first request for voyage ai, so we'll keep this in mind.

luiseok•1/23/25, 4:55 AM

I was just about to request the addition of VoyageAI, too. It would be great if VoyageAI could be added

David•1/24/25, 7:43 AM

hello why do i have cors issues using the ai gateway (localhost origin)

BumblebeeSquare•1/24/25, 10:28 PM

Hi all does vercel ai sdk work with googe ai studio endpoint + ai gateway ? i tried https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/google-ai-studio
but didnt work , i kept getting 404 statu scode on dashboard, after i removed gateway endpoint and use google's endpoint, it work just fine

BumblebeeSquare•1/24/25, 10:30 PM

another question: does gateway support IP based rate limiting

David•1/25/25, 11:52 AM

noone here bro

David•1/25/25, 11:52 AM

:NotLikeThis:

nukely•1/26/25, 4:11 PM

How to handle user privacy and data security in ai gateway caching?

Nnukely How to handle user privacy and data security in ai gateway caching?

Isaac McFadyen•1/26/25, 4:28 PM

https://developers.cloudflare.com/ai-gateway/configuration/caching/#custom-cache-key-cf-aig-cache-key Use a custom Cache Key of some ID or similar that represents your user.

Isaac McFadyen•1/26/25, 4:28 PM

Then cache won't be shared across users.

nukely•1/26/25, 5:19 PM

Thanks

BBumblebeeSquare another question: does gateway support IP based rate limiting

Kathy•1/27/25, 8:16 PM

not at the moment. our rate limits right now are by number of requests through a gateway https://developers.cloudflare.com/ai-gateway/configuration/rate-limiting/

if we had rate limiting based on custom metadata, would that work for your use case?

Cloudflare Docs

Rate limiting · Cloudflare AI Gateway docs

Rate limiting controls the traffic that reaches your application, which prevents expensive bills and suspicious activity.

KKathy not at the moment. our rate limits right now are by number of requests through a...

BumblebeeSquare•1/27/25, 9:43 PM

yeah definitely, as i can just add ip to metadata, currently i use durable object to rate limit user to 1 req/15s, not sure which is more cost effective tho. It would be great if i can also do filter on token cost by metadata, i.e i want to know how much ishould charge a user by user_id

BumblebeeSquare•1/27/25, 9:50 PM

btw the cost for gemini flash looks diffferent from my calculation: 415 * 0.075 / 1m+ 36*0.3 / 1million = 0.00004129, is my calc wrong ....

kata 1000•1/28/25, 3:05 AM

Hi everyone, can the gateways be configured to protect my custom serverless function? Currently, it seems they are only set up for direct connections to various mainstream AI API providers. I have an AI bot that connects to different databases and prompts, all bundled into a serverless function. The AI gateways are excellent for setting up authentication, rate limiting, and other features. Is there a way to combine them together ?

laurynas•1/28/25, 2:57 PM

Hi hi! Any chance we could add Cerebras provider for AI Gateway?

https://cerebras.ai/

Cerebras

Cerebras is the go-to platform for fast and effortless AI training. Learn more at cerebras.ai.

tornado•1/28/25, 8:52 PM

Any idea why gemini requests aren't being cached or shown in the AI gateway dashboard while I can see claude?

tornado•1/28/25, 10:26 PM

Is there any new bug with gemini logs. I think after I turned on caching, gemini logs stopped showing up in the AI gateway dashboard. It also seems like the cache isn't being applied to them.

Llaurynas Hi hi! Any chance we could add Cerebras provider for AI Gateway? 🙌 https://cere...

Kathy•1/29/25, 12:06 AM

we do take into account requests! thanks for letting me know.

curious why you're using them vs other providers

Ttornado Is there any new bug with gemini logs. I think after I turned on caching, gemini...

Kathy•1/29/25, 12:07 AM

are you using google vertex or google ai studio? i just tested on google ai studio and seems like caching is working properly. Have you set any different cache configuration for gemini? https://developers.cloudflare.com/ai-gateway/configuration/caching/

Cloudflare Docs

Caching · Cloudflare AI Gateway docs

Override caching settings on a per-request basis.

BBumblebeeSquare btw the cost for gemini flash looks diffferent from my calculation: 415 * 0.07...

Kathy•1/29/25, 12:17 AM

thanks for pointing out. ill get that fixed

KKathy we do take into account requests! thanks for letting me know. curious why you'...

laurynas•1/29/25, 6:23 AM

thank you! Experimenting still but mostly due to speed. LLaMa on Cerebras architecture outputs at 2k tokens/sec which makes a world of difference in terms of latency in the UX

BumblebeeSquare•1/29/25, 6:58 PM

Hi guys, is there a way to do analytic, collect token costs without enable logs ? i feel my user prob dont want me to see their inappropriate prompts in logs

KKathy are you using google vertex or google ai studio? i just tested on google ai stud...

tornado•1/29/25, 7:03 PM

I'm using Google AI studio and no changes to default caching. I did more investigation and it seems like when I send request locally things show up on the dashboard but inside my deployed cloudflare workflow only the Claude api calls show up in the dashboard.

rob•1/30/25, 11:24 PM

https://x.com/ritakozlov_/status/1885034425538187610

rita kozlov 🐀 (@ritakozlov_) on X

we shipped a binding you can use for calling @cloudflaredev ai gateway directly from a worker! if you're already using workers + workers ai, just add this to your existing code:

gateway: {
id: "my-gateway"
}
or send granular feedback w your logs →
https://t.co/ZZIvA04kPb

•

1/30/25, 6:36 PM

Rrob https://x.com/ritakozlov_/status/1885034425538187610

Kathy•1/30/25, 11:35 PM

hope yall like!

Zig•1/31/25, 12:42 AM

Does anyone know how to use the websockets api with streaming? Im trying to use the universal endpoint with openai.

If i send a content type application/json header with no streaming it works fine, if i try and do it with streaming i see the request going through on the dashboard but my worker never receives any websocket messages. if i send it without a content type header or with event-stream i allways get an error about a missing model

KKathy what model are you using from anthropic? i can check

dom•2/3/25, 2:06 PM

Using claude-3-5-sonnet-latest which currently defaults to claude-3-5-sonnet-20241022

I see o3 mini also has this issue.

usualdevOP•2/5/25, 9:02 PM

Feature request, add ability to adjust the column length. I don't want feedback, latency, or status to be that wide, I would rather see the model name

VIO•2/6/25, 2:18 PM

url not working https://developers.cloudflare.com/ai-gateway/request-handling/

VVIO url not working https://developers.cloudflare.com/ai-gateway/request-handling/

Kathy•2/6/25, 7:46 PM

how did you get to this url?
correct one is: https://developers.cloudflare.com/ai-gateway/configuration/request-handling/

Cloudflare Docs

Request handling · Cloudflare AI Gateway docs

Your AI gateway supports different strategies for handling requests to providers, which allows you to manage AI interactions effectively and ensure your applications remain responsive and reliable.

Kathy•2/6/25, 7:47 PM

nevermind i found it- the changelog THANKS

rob•2/7/25, 2:08 AM

https://x.com/CloudflareDev/status/1887636157963649046

Cloudflare Developers (@CloudflareDev) on X

We've got 2 new exciting updates for AI Gateway!

•Gain observability and control over your Cerebras,
ElevenLabs and Cartesia usage via AI Gateway.

•Add more control to your requests with new timeout, retry, and fallback options.

Learn more about these two updates below

•

2/6/25, 10:55 PM

luiseok•2/7/25, 3:23 AM

Does anyone get errors when trying to google gemini 1.5 flash? I know it's not an ai gateway problem, but just to be sure if I'm the only one having trouble.

rob•2/7/25, 2:25 PM

Kavatch•2/7/25, 4:55 PM

There is no universal way to use ai gateway with any openai compatabil endpoint correct?
I am using deepinfra and would like to use ai gateway however I don't see any options to set my own base url

morpig•2/8/25, 11:13 AM

hi! wondering if there any plans to return request ID (what we see in AI gateway logs) in workers. or is it possible already?

GGabriel const myLogId = env.AI.aiGatewayLogId;

morpig•2/9/25, 12:34 PM

my bad for not looking further! this is awesome. thank you!

BumblebeeSquare•2/10/25, 1:21 AM

hi guys , anyone know why https://github.com/cloudflare/workers-ai-provider does not return promptToken ? it just return zero

GitHub

GitHub - cloudflare/workers-ai-provider: A Workers AI provider for ...

A Workers AI provider for the vercel AI SDK. Contribute to cloudflare/workers-ai-provider development by creating an account on GitHub.

With OpenRouter, I can use the key `models` to send up to 3 models and OpenRouter will pick one base

Similar Threads

Similar Threads

Similar Threads