Cloudflare Developers•9mo ago

Sadly for steaming requests, this is always gonna be a limitation. 😦 It sucks HTTP doesn’t allow am

Sadly for steaming requests, this is always gonna be a limitation.

It sucks HTTP doesn’t allow amending/setting headers after the response body.

Ddave Sadly for steaming requests, this is always gonna be a limitation. 😦 It sucks H...

Isaac McFadyen•4/14/25, 3:16 PM

Except that here it wasn't streamed.

Isaac McFadyen•4/14/25, 3:16 PM

Presumably they have enough information about the response by the time they send the HTTP status that they could change it if there's an error

m4rs0n•4/15/25, 4:12 AM

Subject: Grok API Location Detection Issue When Using ai-gateway from Mainland China

Description:

Hello Cloudflare team,

I am a developer based in Mainland China. I am currently using ai-gateway to call the Grok API. I have noticed that when I make requests directly (without a VPN) from my local developing Mac, Grok is able to detect my actual location and refuse my request. However, when I use a VPN to make the same Grok API calls, this location detection issue does not occur.

Could you please help me understand why Grok is able to recognize my real location when I am not using a VPN, and whether there are any configuration options or best practices to prevent this? Is this expected behavior for users in Mainland China, or is there something specific to ai-gateway or my setup that I should be aware of?

Thank you for your assistance!

cosbgn•4/16/25, 12:12 PM

I have an app where users add their own OpenAI api key. So the Auth header will be different from user to user.

Can I still use

ai-gateway

ai-gateway

or cache will get mixed up?

rufi•4/16/25, 1:41 PM

how can i integrate the ai-gateway with createWorkersAI (workers-ai-provider)?

import { createWorkersAI } from 'workers-ai-provider';
import { streamText } from 'ai';
...
const model = '@cf/meta/llama-3.1-70b-instruct';
    const workersai = createWorkersAI({ binding: env.AI, 
        gateway: {
          id: "ai-gateway-id ?"
        },
        baseURL: 'https://gateway.ai.cloudflare.com/...?'
      });
      ...
    const result = streamText({
      model: workersai(model),
      prompt: finalPrompt,
    });

import { createWorkersAI } from 'workers-ai-provider';
import { streamText } from 'ai';
...
const model = '@cf/meta/llama-3.1-70b-instruct';
    const workersai = createWorkersAI({ binding: env.AI, 
        gateway: {
          id: "ai-gateway-id ?"
        },
        baseURL: 'https://gateway.ai.cloudflare.com/...?'
      });
      ...
    const result = streamText({
      model: workersai(model),
      prompt: finalPrompt,
    });

luiseok•4/17/25, 5:52 AM

AI Gateway is currently returning a 500 internal server error. Am I being affected by API performance degradation?

abuiles•4/19/25, 1:38 AM

Minor nit, but it seems like the logs filter UI is assuming strings in the metadata.

The example in the docs shows this

--header 'cf-aig-metadata: {"team": "AI", "user": 12345, "test":true}' \

--header 'cf-aig-metadata: {"team": "AI", "user": 12345, "test":true}' \

But then when you try to filter in the UI it won't work

filters: [{"key":"metadata.value","operator":"eq","value":["12345"]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

filters: [{"key":"metadata.value","operator":"eq","value":["12345"]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

I changed the request manually to a a number and it worked

filters: [{"key":"metadata.value","operator":"eq","value":[12345]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

filters: [{"key":"metadata.value","operator":"eq","value":[12345]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

Is this expected?

面

面条•4/21/25, 12:57 PM

Cloudflare AI Gateway has a 100-second origin timeout. But when using inference models like Gemini 2.5 Pro without streaming output, it's easy to exceed that 100-second limit.

This results in an HTTP 524 error, which is a real headache.

ksanderer•4/21/25, 2:10 PM

Hi everyone,

I'm wondering if there are plans to support cost estimation for streaming requests in AI Gateway?

We're looking at AI Gateway + Logpush for our SaaS application where we need to track costs across different tenants. Currently, we're using various models from different AI providers, and managing cost tracking has become quite messy.

AI Gateway looks really promising for our use case, but we specifically need to understand how streaming requests would be handled in terms of cost estimation.

Is there a public roadmap available that might indicate when this feature could be expected? Or any information about the development timeline for streaming request cost tracking?

Aabuiles Minor nit, but it seems like the logs filter UI is assuming strings in the metad...

Kathy•4/21/25, 7:39 PM

nice catch - thanks for raising

Kksanderer Hi everyone, I'm wondering if there are plans to support cost estimation for st...

Kathy•4/21/25, 7:39 PM

which models and providers are you using for streaming that do not have tokens?

面面条 Cloudflare AI Gateway has a 100-second origin timeout. But when using inference ...

Kathy•4/21/25, 8:58 PM

fyi this has been raised

mr.niko.la•4/22/25, 7:28 AM

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic({
  apiKey: "<api key>",
  baseURL:
    "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/grok",
});

const msg = await anthropic.messages.create({
  model: "grok-beta",
  max_tokens: 128,
  system:
    "You are Grok, a chatbot inspired by the Hitchhiker's Guide to the Galaxy.",
  messages: [
    {
      role: "user",
      content: "What is the meaning of life, the universe, and everything?",
    },
  ],
});

console.log(msg);

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic({
  apiKey: "<api key>",
  baseURL:
    "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/grok",
});

const msg = await anthropic.messages.create({
  model: "grok-beta",
  max_tokens: 128,
  system:
    "You are Grok, a chatbot inspired by the Hitchhiker's Guide to the Galaxy.",
  messages: [
    {
      role: "user",
      content: "What is the meaning of life, the universe, and everything?",
    },
  ],
});

console.log(msg);

https://developers.cloudflare.com/ai-gateway/providers/grok/

here is the api-key from grok? im kinda confused by the documentation....

I am trying to use grok 3 and grok 3 mini...

Cloudflare Docs

Grok · Cloudflare AI Gateway docs

Grok is s a general purpose model that can be used for a variety of tasks, including generating and understanding text, code, and function calling.

mysterious_dragon_70032•4/22/25, 7:35 AM

#ai-gateway

面面条 Cloudflare AI Gateway has a 100-second origin timeout. But when using inference ...

ItsWendell•4/22/25, 10:01 PM

I've been running into the same problem, I think the 100-second limit also applies for outgoing requests from workers, whether you use AI Gateway or not.

I also did long thinking, non- streaming requests with high output token limits to the Gemini 2.5 APIs where I ran into this problem quite often. I needed to use JSON / structured outputs.

I wrote my engine to use streaming text endpoints. I prompt engineered it in a way where it streams occasional [markers] for real time updates I needed and for structured output I needed for the request, which honestly have been working quite well. It prevents the 100 seconds proxy timeout since the streams start before that time, meanwhile I can comfortably stream up to 40/50k tokens like this using 1 API call.

I also built in some fault tolerance and output recovery, by occasionally storing the output during streaming into KV (Durable Objects storage is also a possibility). If it fails midway and retries in my workflow, it attempts to restore the state and resume where it left off.

What I really like is that I can send real-time updates this way to consumers, since I prompted Gemini to occasionally send an [status: {...}] marker which I can parse in real-time, since outputting 40/50k tokens can time some time.

ItsWendell•4/23/25, 12:59 PM

https://discord.com/channels/595317990191398933/1040420029080018945/1364574169672450112

Awesome that the proxy limit has been increased beyond 100 seconds for AI Gateway, what is the new limit? Just so I can adjust my internal timeouts / retry logic!

ItsWendell•4/23/25, 10:08 PM

@Gabriel nice! Is that only for requests that are

workers <> ai gateway <> model provider

workers <> ai gateway <> model provider

? Or also

workers <> model providers

workers <> model providers

, another case that I'm hitting is that I'm creating context caches at Google Gemini APIs that can take quite a long time, with larger contexts, e.g. a lot of big files, this takes over 100 seconds and I can't use the API gateway for this as far as I know.

IItsWendell @Gabriel nice! Is that only for requests that are `workers <> ai gateway <> mod...

Kathy•4/24/25, 4:21 PM

if requests are not going through AI Gateway, you might need to increase this limit for your domain. Looks like it may be an enterprise only capability https://developers.cloudflare.com/fundamentals/reference/connection-limits/#between-cloudflare-and-origin-server

Cloudflare Docs

Connection limits · Cloudflare Fundamentals docs

When HTTP/HTTPS traffic is proxied through Cloudflare, there are often two established TCP connections: the first is between the requesting client to Cloudflare and the second is between Cloudflare and the origin server. Each connection has their own set of TCP and HTTP limits, which are documented below.

KKathy if requests are not going through AI Gateway, you might need to increase this li...

ItsWendell•4/24/25, 4:41 PM

Thanks for coming back to this! I was aware of the limits and the ability of getting an enterprise plan to upgrade these limits, but that would be for our business in an MVP / prototyping phase not feasable.

If context caching would be supported in AI Gateway, that would be great since than we could also use features like tracking costs of that (in google's case it returns usage, unsure if other providers offer similar methods).

IItsWendell Thanks for coming back to this! I was aware of the limits and the ability of get...

Kathy•4/24/25, 4:49 PM

explain more what you meant by your last sentence on context caching? initially i thought that was re: semantic caching, but seems you're talking about something different

ItsWendell•4/24/25, 5:02 PM

@Kathy | Browser Rendering PM Ah fair, it's related to "context caching" at Google / Vertex AI. For huge contexts and long repeated runs with the same context this can siginificantly speed up inference and cost savings. https://ai.google.dev/gemini-api/docs/caching?lang=rest.

The problem I'm having here is that Google is taking an increasingly long time to more context you give it to create this Context Cache. We often hit 2+ minutes for bigger context (20/30+ PDF files, some of them 100+ pages, which on their side I think are being transformed into images, and then into tokens).

Google AI for Developers

Context caching | Gemini API | Google AI for Developers

Learn how to use Context Caching in the Gemini API

Erisa•4/24/25, 9:45 PM

Is it expected or a bug that requesting workers ai openai-compatible api thru ai gateway (specifically with gemma3 in this case) would not log the request and only the response as shown in the screenshot?

IItsWendell @Kathy | Browser Rendering PM Ah fair, it's related to "context caching" at Goog...

Kathy•4/24/25, 11:49 PM

thanks for sharing so the ask here is for ai gateway to support tracking the costs of context caching through providers, including google

what does the response look like from google when using conext caching? Curious to see how it splits out input, output, and context caching because that is how we would track tokens to then calculate

EErisa Is it expected or a bug that requesting workers ai openai-compatible api thru ai...

Kathy•4/24/25, 11:49 PM

thisssss seems like a bug. could you share your account id + gateway name through DM please

KKathy thisssss seems like a bug. could you share your account id + gateway name throug...

mr.niko.la•4/26/25, 10:04 AM

Hi will you share whisper turbo sample code to test it. Could use a repo with examples.

xpawlik92x•4/26/25, 5:12 PM

Hey!

I'm currently checking AI Gateway for the company I work at — mainly to track token usage per our customer.

While testing GPT-4.1 with streaming enabled, I noticed that if a client cancels a request mid-stream, there’s no trace of that request in the AI Gateway logs.
Is this the intended behavior, or could this be a bug or a limitation of the current implementation?

Azuredush•5/1/25, 10:23 AM

When using Aistudio's OpenAI-compatible endpoint through AI Gateway, a lot of "��" characters appear when the model outputs long content. However, when I use the official endpoint directly, everything works fine. Why is this happening?

AAzuredush When using Aistudio's OpenAI-compatible endpoint through AI Gateway, a lot of "�...

Isaac McFadyen•5/1/25, 4:16 PM

That's probabably not long content but because it's a UTF-8 character that can't be rendered on the Cloudflare dashboard because it's missing in the font they use.

rob•5/2/25, 2:16 PM

daveOP•5/6/25, 10:16 PM

Does AI Gateway retry requests automatically?

daveOP•5/6/25, 10:17 PM

I'm seeing multiple requests logged on OpenAI that I don't see logged in AI Gateway, nor did I send intentionally.

�

😈 Donkey 💫•5/7/25, 11:46 AM

Hello !
What if the endpoint is router? like this:

https://router.huggingface.co/novita/v3/openai/chat/completions

https://router.huggingface.co/novita/v3/openai/chat/completions

mongj•5/9/25, 7:52 PM

hey we're seeing all of our openai STT requests proxied through CF AI gateway fail with 400

{
  "error": {
    "message": "The audio file could not be decoded or its format is not supported.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

{
  "error": {
    "message": "The audio file could not be decoded or its format is not supported.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

Same request work when the original url

https://api.openai.com/v1/audio/transcriptions

https://api.openai.com/v1/audio/transcriptions

is used instead

is anyone else having this problem?

ItsWendell•5/10/25, 1:42 PM

None of the newer gemini models (2.5 Flash / Pro) and even 2.0 Flash (and 2.0 Flash Lite) properly track input / output tokens. We started sending metadata and pricing information into the AI Gateway, but just realized that tokens are not tracked and we don't have control over that.

Google does provide in both streaming and non-streaming use-cases the used tokens in detail, also visible in the output json of AI Gateway.

Couple of Log IDs: 01JTX5HT6G2VCPXCZQAWJ5X39W, 01JTX5JZS78V9Y2XDE8A4WAZ8F, 01JTVQBX08S1WA6DZ0GVS8CV2K

abufoysal2004•5/12/25, 10:10 AM

Yes

Mmongj hey we're seeing all of our openai STT requests proxied through CF AI gateway fa...

kyoya•5/16/25, 2:02 AM

Hello! I'm also having this problem. Hopefully CF can give us an answer

mr.niko.la•5/16/25, 10:02 AM

Which model is the beta for structured outpu? Lllama 4 is

Mmr.niko.la Which model is the beta for structured outpu? Lllama 4 is 😵‍💫🥴

Hygi•5/19/25, 5:31 AM

Hello , i want to ask . So i use the ai gateway its use model llama-4-scout-17b-16e-instruct . So , if i test , i get error like this on my dashboard
{
"errors": [
{
"message": "AiError: AiError: unknown internal error (e06783e9-9892-4065-853d-849bf7b684a9)",
"code": "7000"
}
],
"success": false,
"result": {},
"messages": []
}

why this error ?

ac•5/19/25, 5:45 PM

Hi -- are there plans to add more evaluations to the AI gateway any time soon? e.g. the ability to run model graded evals, similar to the evals that OpenAI provides? This would be huge for us.

Razmjoo•5/22/25, 9:43 AM

Hi there; are there any plans for Google OpenAI-compatible endpoints? Google AI Studio https://ai.google.dev/gemini-api/docs/openai Vertex AI: https://cloud.google.com/vertex-ai/docs/reference/rest

Google AI for Developers

OpenAI compatibility | Gemini API | Google AI for Developers

Google Cloud

Vertex AI API | Google Cloud

import { createWorkersAI } from 'workers-ai-provider'; import { streamText } from 'ai'; ... const model = '@cf/meta/llama-3.1-70b-instruct'; const workersai = createWorkersAI({ binding: env.AI, gateway: { id: "ai-gateway-id ?" }, baseURL: 'https://gateway.ai.cloudflare.com/...?' }); ... const result = streamText({ model: workersai(model), prompt: finalPrompt, });

import Anthropic from "@anthropic-ai/sdk"; const anthropic = new Anthropic({ apiKey: "<api key>", baseURL: "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/grok", }); const msg = await anthropic.messages.create({ model: "grok-beta", max_tokens: 128, system: "You are Grok, a chatbot inspired by the Hitchhiker's Guide to the Galaxy.", messages: [ { role: "user", content: "What is the meaning of life, the universe, and everything?", }, ], }); console.log(msg);

Sadly for steaming requests, this is always gonna be a limitation. 😦 It sucks HTTP doesn’t allow am

Similar Threads

Similar Threads

Similar Threads