ModelContextProtocol

I

ItsWendell•5/10/25, 1:42 PM

None of the newer gemini models (2.5 Flash / Pro) and even 2.0 Flash (and 2.0 Flash Lite) properly track input / output tokens. We started sending metadata and pricing information into the AI Gateway, but just realized that tokens are not tracked and we don't have control over that.

Google does provide in both streaming and non-streaming use-cases the used tokens in detail, also visible in the output json of AI Gateway.

Couple of Log IDs: 01JTX5HT6G2VCPXCZQAWJ5X39W, 01JTX5JZS78V9Y2XDE8A4WAZ8F, 01JTVQBX08S1WA6DZ0GVS8CV2K

A

abufoysal2004•5/12/25, 10:10 AM

Yes

Mmongj hey we're seeing all of our openai STT requests proxied through CF AI gateway fa...

K

kyoya•5/16/25, 2:02 AM

Hello! I'm also having this problem. Hopefully CF can give us an answer

M

mr.niko.la•5/16/25, 10:02 AM

Which model is the beta for structured outpu? Lllama 4 is

Mmr.niko.la Which model is the beta for structured outpu? Lllama 4 is 😵‍💫🥴

H

Hygi•5/19/25, 5:31 AM

Hello , i want to ask . So i use the ai gateway its use model llama-4-scout-17b-16e-instruct . So , if i test , i get error like this on my dashboard
{
"errors": [
{
"message": "AiError: AiError: unknown internal error (e06783e9-9892-4065-853d-849bf7b684a9)",
"code": "7000"
}
],
"success": false,
"result": {},
"messages": []
}

why this error ?

A

ac•5/19/25, 5:45 PM

Hi -- are there plans to add more evaluations to the AI gateway any time soon? e.g. the ability to run model graded evals, similar to the evals that OpenAI provides? This would be huge for us.

R

Razmjoo•5/22/25, 9:43 AM

Hi there; are there any plans for Google OpenAI-compatible endpoints? Google AI Studio https://ai.google.dev/gemini-api/docs/openai Vertex AI: https://cloud.google.com/vertex-ai/docs/reference/rest

Google AI for Developers

OpenAI compatibility | Gemini API | Google AI for Developers

Google Cloud

Vertex AI API | Google Cloud

R

Razmjoo•5/22/25, 9:48 AM

I am unsure if this is the best place for a bug report... AI Gateway openai comptable for workers-ai (https://developers.cloudflare.com/ai-gateway/providers/workersai/#openai-compatible-endpoints) isn't working properly when sending an image to llama4 (https://developers.cloudflare.com/workers-ai/models/llama-4-scout-17b-16e-instruct/) while if using AI Gateway + Direct model call (@cf/meta/llama-4-scout-17b-16e-instruct) it will work.

Cloudflare Docs

llama-4-scout-17b-16e-instruct

Meta's Llama 4 Scout is a 17 billion parameter model with 16 experts that is natively multimodal. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.

Cloudflare Docs

Workers AI

Use AI Gateway for analytics, caching, and security on requests to Workers AI. Workers AI integrates seamlessly with AI Gateway, allowing you to execute AI inference via API requests or through an environment binding for Workers scripts. The binding simplifies the process by routing requests through your AI Gateway with minimal setup.

Y

yinxingmaiming6409•6/1/25, 2:03 PM

Is there a way to obtain the image embedding on CF?

N

Nico•6/4/25, 12:19 AM

I'm trying out the new OpenAI-compatible endpoint and running into a weird difference between using the OpenAI SDK and curl. I've tested the exact code from the docs for both methods and curl works but the typescript example does not.

Here's the typescript:

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.GEMINI_API_KEY,
    baseURL: "<my gateway url>"
});

const response = await client.chat.completions.create({
    model: "google-ai-studio/gemini-2.0-flash",
    messages: [{ role: "user", content: "What is Cloudflare?" }],
});

console.log(response.choices[0]?.message.content);

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.GEMINI_API_KEY,
    baseURL: "<my gateway url>"
});

const response = await client.chat.completions.create({
    model: "google-ai-studio/gemini-2.0-flash",
    messages: [{ role: "user", content: "What is Cloudflare?" }],
});

console.log(response.choices[0]?.message.content);

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.GEMINI_API_KEY,
    baseURL: "<my gateway url>"
});

const response = await client.chat.completions.create({
    model: "google-ai-studio/gemini-2.0-flash",
    messages: [{ role: "user", content: "What is Cloudflare?" }],
});

console.log(response.choices[0]?.message.content);

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.GEMINI_API_KEY,
    baseURL: "<my gateway url>"
});

const response = await client.chat.completions.create({
    model: "google-ai-studio/gemini-2.0-flash",
    messages: [{ role: "user", content: "What is Cloudflare?" }],
});

console.log(response.choices[0]?.message.content);

This code responds with a 500 erorr:

error: 500 [{"code":2002,"message":"Internal server error"}]
    status: 500,
   headers: Headers {
  "date": "Wed, 04 Jun 2025 00:17:33 GMT",
  "content-type": "application/json",
  "content-length": "101",
  "connection": "keep-alive",
  "vary": "Accept-Encoding",
  "set-cookie": [ "__cf_bm=lxBSjxIyTfBLOugJfEcUnFQKJkr6p7.457_228mHwaY-1748996253-1.0.1.1-X7FS2Pm_wVKIaxeYigqVUhnTxNN9iz.UjHnzyc1mlSb.XVfZET4Qu.h1ipuCSntWPgT3rlOGusU1gjNJAKHq_SKTPaueVJM6jp6n4pkt.kU; path=/; expires=Wed, 04-Jun-25 00:47:33 GMT; domain=.gateway.ai.cloudflare.com; HttpOnly; Secure; SameSite=None" ],
  "server": "cloudflare",
  "cf-ray": "94a33f75f8e2000e-ORD",
},
 requestID: null,
     error: [
  [Object ...]
],

error: 500 [{"code":2002,"message":"Internal server error"}]
    status: 500,
   headers: Headers {
  "date": "Wed, 04 Jun 2025 00:17:33 GMT",
  "content-type": "application/json",
  "content-length": "101",
  "connection": "keep-alive",
  "vary": "Accept-Encoding",
  "set-cookie": [ "__cf_bm=lxBSjxIyTfBLOugJfEcUnFQKJkr6p7.457_228mHwaY-1748996253-1.0.1.1-X7FS2Pm_wVKIaxeYigqVUhnTxNN9iz.UjHnzyc1mlSb.XVfZET4Qu.h1ipuCSntWPgT3rlOGusU1gjNJAKHq_SKTPaueVJM6jp6n4pkt.kU; path=/; expires=Wed, 04-Jun-25 00:47:33 GMT; domain=.gateway.ai.cloudflare.com; HttpOnly; Secure; SameSite=None" ],
  "server": "cloudflare",
  "cf-ray": "94a33f75f8e2000e-ORD",
},
 requestID: null,
     error: [
  [Object ...]
],

error: 500 [{"code":2002,"message":"Internal server error"}]
    status: 500,
   headers: Headers {
  "date": "Wed, 04 Jun 2025 00:17:33 GMT",
  "content-type": "application/json",
  "content-length": "101",
  "connection": "keep-alive",
  "vary": "Accept-Encoding",
  "set-cookie": [ "__cf_bm=lxBSjxIyTfBLOugJfEcUnFQKJkr6p7.457_228mHwaY-1748996253-1.0.1.1-X7FS2Pm_wVKIaxeYigqVUhnTxNN9iz.UjHnzyc1mlSb.XVfZET4Qu.h1ipuCSntWPgT3rlOGusU1gjNJAKHq_SKTPaueVJM6jp6n4pkt.kU; path=/; expires=Wed, 04-Jun-25 00:47:33 GMT; domain=.gateway.ai.cloudflare.com; HttpOnly; Secure; SameSite=None" ],
  "server": "cloudflare",
  "cf-ray": "94a33f75f8e2000e-ORD",
},
 requestID: null,
     error: [
  [Object ...]
],

error: 500 [{"code":2002,"message":"Internal server error"}]
    status: 500,
   headers: Headers {
  "date": "Wed, 04 Jun 2025 00:17:33 GMT",
  "content-type": "application/json",
  "content-length": "101",
  "connection": "keep-alive",
  "vary": "Accept-Encoding",
  "set-cookie": [ "__cf_bm=lxBSjxIyTfBLOugJfEcUnFQKJkr6p7.457_228mHwaY-1748996253-1.0.1.1-X7FS2Pm_wVKIaxeYigqVUhnTxNN9iz.UjHnzyc1mlSb.XVfZET4Qu.h1ipuCSntWPgT3rlOGusU1gjNJAKHq_SKTPaueVJM6jp6n4pkt.kU; path=/; expires=Wed, 04-Jun-25 00:47:33 GMT; domain=.gateway.ai.cloudflare.com; HttpOnly; Secure; SameSite=None" ],
  "server": "cloudflare",
  "cf-ray": "94a33f75f8e2000e-ORD",
},
 requestID: null,
     error: [
  [Object ...]
],

The equivalent curl request works exactly as expected

curl -X POST "<my gateway url>" \
  --header "Authorization: Bearer $GEMINI_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "google-ai-studio/gemini-2.0-flash",
    "messages": [
      {
        "role": "user",
        "content": "What is Cloudflare?"
      }
    ]
  }'

curl -X POST "<my gateway url>" \
  --header "Authorization: Bearer $GEMINI_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "google-ai-studio/gemini-2.0-flash",
    "messages": [
      {
        "role": "user",
        "content": "What is Cloudflare?"
      }
    ]
  }'

curl -X POST "<my gateway url>" \
  --header "Authorization: Bearer $GEMINI_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "google-ai-studio/gemini-2.0-flash",
    "messages": [
      {
        "role": "user",
        "content": "What is Cloudflare?"
      }
    ]
  }'

curl -X POST "<my gateway url>" \
  --header "Authorization: Bearer $GEMINI_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "google-ai-studio/gemini-2.0-flash",
    "messages": [
      {
        "role": "user",
        "content": "What is Cloudflare?"
      }
    ]
  }'

Cloudflare Docs

OpenAI Compatibility

Cloudflare's AI Gateway offers an OpenAI-compatible /chat/completions endpoint, enabling integration with multiple AI providers using a single URL. This feature simplifies the integration process, allowing for seamless switching between different models without significant code modifications.

面

面条•6/4/25, 2:25 AM

Is the OpenAI compatible endpoint not supporting streaming output?

N

Nico•6/4/25, 3:06 AM

Maybe?

N

Nico•6/4/25, 3:06 AM

But also the curl didn't stream it's responsd

NNico But also the curl didn't stream it's responsd

I

Isaac McFadyen•6/4/25, 3:18 PM

cURL does if you specify the

-N

-N

(nobuffer) flag

IIsaac McFadyen cURL does if you specify the `-N` (nobuffer) flag

N

Nico•6/4/25, 3:23 PM

Thanks! Didn't know this. Behavior didn't change

I

Isaac McFadyen•6/4/25, 3:23 PM

Did you also specify

"stream": true

"stream": true

?

N

Nico•6/4/25, 3:24 PM

yes

I

Isaac McFadyen•6/4/25, 3:24 PM

Interesting, maybe it doesn't support it then.

N

Nico•6/4/25, 3:24 PM

Yeah must be. Curious what the error with the sdk version is

NNico Yeah must be. Curious what the error with the sdk version is

I

Isaac McFadyen•6/4/25, 3:30 PM

When you specified the URL with the SDK what URL did you use? IIRC the OpenAI SDK automatically adds

/chat/completion

/chat/completion

to the end, so your URL shouldn't have that in it with the SDK

NNico Yeah must be. Curious what the error with the sdk version is

N

Nico•6/4/25, 3:32 PM

I used

https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/compat/chat/completions

https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/compat/chat/completions

which is what is in the docs. But I'll try it without

/chat/completions

/chat/completions

!

N

Nico•6/4/25, 3:33 PM

Yes that was the issue. The docs are wrong.

I

Isaac McFadyen•6/4/25, 3:47 PM

I'll submit a PR - thanks.

C

camudo•6/4/25, 4:22 PM

Are you guys able to see the AI Gateway logs normally?
general-helpAI Gateway bug?

C

camudo•6/4/25, 4:23 PM

ok seems out

Captura_de_Tela_2025-06-04_as_13.23.11.png

A

aflores•6/4/25, 6:17 PM

does llm streaming not work with the byob api keys way of using api gateway? kind of unusable for a lot of front facing client work if i can't stream an api response with my own keys and instead have to use some workers ai thing with the websocket?
don't see clarification in the docs and quick search here didn't turn up much. i can set streaming true in the grok version of the call and see it coming in as chucnks but it buffers them all before sending them down as a single response

�

😈 Donkey 💫•6/4/25, 6:32 PM

AI gateway failed to render data

�

😈 Donkey 💫•6/4/25, 6:32 PM

Invalid date and no content

�😈 Donkey 💫AI gateway failed to render data

K

Kathy•6/4/25, 9:46 PM

thanks for reporting! we've fixed this bug

KKathy thanks for reporting! we've fixed this bug

�

😈 Donkey 💫•6/6/25, 9:02 AM

Hello,

Now, Vertex AI accept location_id with "global" instead of specific regions like the us-central1... I can call with endpoint aiplatform.googleapis.com

But I can't call "global" via AI Gateway, it throws 404

D

dave•6/6/25, 5:08 PM

Could we have support for the new

https://*.openai.azure.com/openai/v1/*

https://*.openai.azure.com/openai/v1/*

Azure OpenAI endpoint? I can't seem to get it working with AI Gateway; I'm guessing something is being mutated on the AI Gateway side incorrectly?

https://learn.microsoft.com/en-us/azure/ai-services/openai/reference-preview-latest

D

DrWayne•6/6/25, 5:18 PM

FYI, I was using the non-streaming websockets interface yesterday and the docs just say "Alternatively, we also support authentication via the sec-websocket-protocol header if you are using a browser WebSocket." without telling how. Fortunately, I found a snippet in a blog post about DO and websockets.

cf-aig-authorization.${AI_GATEWAY_TOKEN}

cf-aig-authorization.${AI_GATEWAY_TOKEN}

It would be nice if that was explicitly in the docs since it is non-obvious how to form that sec-websocket-protocol header to encapsulate the cf-aig-authorization header we'dve used in Node.js instead of the browser...

B

Bryan Chen•6/8/25, 9:34 AM

AI Gateway Universal Endpoint stops working for me. It seems like the auth header failed to be passed and always getting auth error from providers

�😈 Donkey 💫Hello, Now, Vertex AI accept location_id with "global" instead of specific regi...

O

Oblomki•6/10/25, 6:04 PM

Same problem. AI Gateway seems to replace 'global' with a region, leading to a 404 error.

O

Oli•6/13/25, 12:36 PM

Question for yall who are more experienced with ai gateway,

Were calling anthropic,openai,azure,grok,groq,deepseek:s sse streaming endpoints serverside, so essentially, its only called after any rate-limiting, auth, etc.

But ive been looking at the ai gateway to see if there would be any benefit to calling the providers trough ai gateway,
Would there be any benefit to calling those api:s trough ai gateway? if the calls are user-independant, and with rate-limiting essentially disabled?

How does the caching work?
Is it just simple like if the exact same prompt is sent twice it will be a cache hit? or does it work with variations of the same prompt etc?

Also, does it handle retries etc?

--
Question 2:
We serve all models from all providers over ws to client.
Is it possible with ai gateway, to instead of serverside > ai-gateway > aiprovider. Somehow do:
client > aiprovider > our-backend ?
Using a custom api?

D

dave•6/13/25, 8:02 PM

Would be nice if we could get token pricing for a bunch of the new models from Anthropic and OpenAI

M

Marino•6/14/25, 2:49 PM

Does the universal endpoint support tools?

Sspongebhav I didn’t fully understand your question. By tools do you mean MCPs? or if we can...

M

Marino•6/14/25, 5:36 PM

I mean tools as in function calling, I couldn't find anything in the docs.

MMarino Does the universal endpoint support tools?

U

U.Senthil Kumar | Contributor•6/15/25, 7:13 AM

Yes, the Universal Endpoint supports tools by allowing multi-provider payloads with specific configurations per message. It also handles retries and fallback logic, making tool use more flexible across providers.

UU.Senthil Kumar | Contributor Yes, the Universal Endpoint supports tools by allowing multi-provider payloads w...

M

Marino•6/15/25, 2:59 PM

Thank you.

hey we're seeing all of our openai STT requests proxied through CF AI gateway fail with 400 ``` {

Similar Threads

Similar Threads

Similar Threads