Could you tell me how long it will take for Llama 4 to be integrated?

akshay•4/6/25, 2:37 PM

Hey folks!

I'm currently exploring how to build with OpenAI's real-time API and Cloudflare. Most examples I've come across are based on WebSockets, but I'm considering using WebRTC instead due to potential performance benefits.

Does anyone have references or examples of using WebRTC in this context? Would love to learn from any prior work or suggestions. Thanks in advance!

VVladimir_dev Could you tell me how long it will take for Llama 4 to be integrated?

Kathy•4/7/25, 11:02 PM

Are you referring to Workers AI? If so, that was added yesterday https://blog.cloudflare.com/meta-llama-4-is-now-available-on-workers-ai/
there's a different channel for Workers AI workers-ai

The Cloudflare Blog

Meta Llama 4 now available on Workers AI

Llama 4 Scout 17B Instruct is now available on Workers AI: use this multimodal, Mixture of Experts AI model on Cloudflare's serverless AI platform to build next-gen AI applications. Build products that can understand images & text, accept large context, with high performance on model quality and inference latency.

Aakshay Hey folks! 👋 I'm currently exploring how to build with OpenAI's real-time API a...

Kathy•4/7/25, 11:03 PM

webRTC not yet supported in AI Gateway but will in the future. In the meantime, can check out Calls https://developers.cloudflare.com/calls/

Cloudflare Docs

Overview · Cloudflare Calls docs

Cloudflare Calls is infrastructure for real-time audio/video/data applications. It allows you to build real-time apps without worrying about scaling or regions. It can act as a selective forwarding unit (WebRTC SFU), as a fanout delivery system for broadcasting (WebRTC CDN) or anything in between.

smile•4/8/25, 4:03 AM

Hi all, I've been trying to work with the Cloudflare WebSocket API, but it isn't functioning as expected. I've reached out to the Cloudflare community and Discord for support, but unfortunately, I haven’t received any response so far.

Could someone please help me identify whether the issue lies with the Cloudflare API itself or if there's something wrong in my implementation?

Jaka•4/8/25, 5:08 AM

i have AutoRag running from yesterday. nothing seems to happen. Things appear blocked. Need help.

Bryan Chen•4/8/25, 8:40 AM

Any plan to improve the logs viewer? Currently the requests and responses are just rendered as JSON, which makes it impossible to read (due to

\n

\n

s). I am really considering build a 3rd party viewer for it.

Original message was deleted

smile•4/9/25, 8:07 AM

Hi, Did you use Webscoker API in cloudflare?

Radek D.•4/9/25, 4:46 PM

hi all I got stuck on this I want to call from AI Agent to AI gateway via binding env.AI but can I call public providers OpenAI/Anthropic etc or only CF models ? https://developers.cloudflare.com/agents/api-reference/using-ai-models/#model-routing

Cloudflare Docs

Using AI Models · Cloudflare Agents docs

Agents can communicate with AI models hosted on any provider, including:

Radek D.•4/9/25, 4:47 PM

if I want to call public providers I need to call AI Gatewat directly via HTTP or wss or can use env.AI binding ? apreciate some help

Radek D.•4/9/25, 5:10 PM

https://developers.cloudflare.com/ai-gateway/integrations/worker-binding-methods/ found it !

Cloudflare Docs

AI Gateway Binding Methods · Cloudflare AI Gateway docs

This guide provides an overview of how to use the latest Cloudflare Workers AI Gateway binding methods. You will learn how to set up an AI Gateway binding, access new methods, and integrate them into your Workers.

AAppSail Hi, Why did this happen?

Kathy•4/11/25, 5:45 PM

this is an issue with OpenRouter returning an error but with a 200 http code

JJaka i have AutoRag running from yesterday. nothing seems to happen. Things appear bl...

Kathy•4/11/25, 5:45 PM

ai-search

AAppSail Hi, Why did this happen?

Kathy•4/11/25, 5:46 PM

Could you send me your account info please? we can make a fix for this

KKathy this is an issue with OpenRouter returning an error but with a 200 http code

dave•4/14/25, 11:02 AM

Sadly for steaming requests, this is always gonna be a limitation.

It sucks HTTP doesn’t allow amending/setting headers after the response body.

Ddave Sadly for steaming requests, this is always gonna be a limitation. 😦 It sucks H...

Isaac McFadyen•4/14/25, 3:16 PM

Except that here it wasn't streamed.

Isaac McFadyen•4/14/25, 3:16 PM

Presumably they have enough information about the response by the time they send the HTTP status that they could change it if there's an error

m4rs0n•4/15/25, 4:12 AM

Subject: Grok API Location Detection Issue When Using ai-gateway from Mainland China

Description:

Hello Cloudflare team,

I am a developer based in Mainland China. I am currently using ai-gateway to call the Grok API. I have noticed that when I make requests directly (without a VPN) from my local developing Mac, Grok is able to detect my actual location and refuse my request. However, when I use a VPN to make the same Grok API calls, this location detection issue does not occur.

Could you please help me understand why Grok is able to recognize my real location when I am not using a VPN, and whether there are any configuration options or best practices to prevent this? Is this expected behavior for users in Mainland China, or is there something specific to ai-gateway or my setup that I should be aware of?

Thank you for your assistance!

cosbgn•4/16/25, 12:12 PM

I have an app where users add their own OpenAI api key. So the Auth header will be different from user to user.

Can I still use

ai-gateway

ai-gateway

or cache will get mixed up?

rufi•4/16/25, 1:41 PM

how can i integrate the ai-gateway with createWorkersAI (workers-ai-provider)?

import { createWorkersAI } from 'workers-ai-provider';
import { streamText } from 'ai';
...
const model = '@cf/meta/llama-3.1-70b-instruct';
    const workersai = createWorkersAI({ binding: env.AI, 
        gateway: {
          id: "ai-gateway-id ?"
        },
        baseURL: 'https://gateway.ai.cloudflare.com/...?'
      });
      ...
    const result = streamText({
      model: workersai(model),
      prompt: finalPrompt,
    });

import { createWorkersAI } from 'workers-ai-provider';
import { streamText } from 'ai';
...
const model = '@cf/meta/llama-3.1-70b-instruct';
    const workersai = createWorkersAI({ binding: env.AI, 
        gateway: {
          id: "ai-gateway-id ?"
        },
        baseURL: 'https://gateway.ai.cloudflare.com/...?'
      });
      ...
    const result = streamText({
      model: workersai(model),
      prompt: finalPrompt,
    });

luiseok•4/17/25, 5:52 AM

AI Gateway is currently returning a 500 internal server error. Am I being affected by API performance degradation?

abuiles•4/19/25, 1:38 AM

Minor nit, but it seems like the logs filter UI is assuming strings in the metadata.

The example in the docs shows this

--header 'cf-aig-metadata: {"team": "AI", "user": 12345, "test":true}' \

--header 'cf-aig-metadata: {"team": "AI", "user": 12345, "test":true}' \

But then when you try to filter in the UI it won't work

filters: [{"key":"metadata.value","operator":"eq","value":["12345"]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

filters: [{"key":"metadata.value","operator":"eq","value":["12345"]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

I changed the request manually to a a number and it worked

filters: [{"key":"metadata.value","operator":"eq","value":[12345]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

filters: [{"key":"metadata.value","operator":"eq","value":[12345]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

Is this expected?

面

面条•4/21/25, 12:57 PM

Cloudflare AI Gateway has a 100-second origin timeout. But when using inference models like Gemini 2.5 Pro without streaming output, it's easy to exceed that 100-second limit.

This results in an HTTP 524 error, which is a real headache.

ksanderer•4/21/25, 2:10 PM

Hi everyone,

I'm wondering if there are plans to support cost estimation for streaming requests in AI Gateway?

We're looking at AI Gateway + Logpush for our SaaS application where we need to track costs across different tenants. Currently, we're using various models from different AI providers, and managing cost tracking has become quite messy.

AI Gateway looks really promising for our use case, but we specifically need to understand how streaming requests would be handled in terms of cost estimation.

Is there a public roadmap available that might indicate when this feature could be expected? Or any information about the development timeline for streaming request cost tracking?

Aabuiles Minor nit, but it seems like the logs filter UI is assuming strings in the metad...

Kathy•4/21/25, 7:39 PM

nice catch - thanks for raising

Kksanderer Hi everyone, I'm wondering if there are plans to support cost estimation for st...

Kathy•4/21/25, 7:39 PM

which models and providers are you using for streaming that do not have tokens?

面面条 Cloudflare AI Gateway has a 100-second origin timeout. But when using inference ...

Kathy•4/21/25, 8:58 PM

fyi this has been raised

mr.niko.la•4/22/25, 7:28 AM

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic({
  apiKey: "<api key>",
  baseURL:
    "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/grok",
});

const msg = await anthropic.messages.create({
  model: "grok-beta",
  max_tokens: 128,
  system:
    "You are Grok, a chatbot inspired by the Hitchhiker's Guide to the Galaxy.",
  messages: [
    {
      role: "user",
      content: "What is the meaning of life, the universe, and everything?",
    },
  ],
});

console.log(msg);

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic({
  apiKey: "<api key>",
  baseURL:
    "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/grok",
});

const msg = await anthropic.messages.create({
  model: "grok-beta",
  max_tokens: 128,
  system:
    "You are Grok, a chatbot inspired by the Hitchhiker's Guide to the Galaxy.",
  messages: [
    {
      role: "user",
      content: "What is the meaning of life, the universe, and everything?",
    },
  ],
});

console.log(msg);

https://developers.cloudflare.com/ai-gateway/providers/grok/

here is the api-key from grok? im kinda confused by the documentation....

I am trying to use grok 3 and grok 3 mini...

Cloudflare Docs

Grok · Cloudflare AI Gateway docs

Grok is s a general purpose model that can be used for a variety of tasks, including generating and understanding text, code, and function calling.

mysterious_dragon_70032•4/22/25, 7:35 AM

#ai-gateway

面面条 Cloudflare AI Gateway has a 100-second origin timeout. But when using inference ...

ItsWendell•4/22/25, 10:01 PM

I've been running into the same problem, I think the 100-second limit also applies for outgoing requests from workers, whether you use AI Gateway or not.

I also did long thinking, non- streaming requests with high output token limits to the Gemini 2.5 APIs where I ran into this problem quite often. I needed to use JSON / structured outputs.

I wrote my engine to use streaming text endpoints. I prompt engineered it in a way where it streams occasional [markers] for real time updates I needed and for structured output I needed for the request, which honestly have been working quite well. It prevents the 100 seconds proxy timeout since the streams start before that time, meanwhile I can comfortably stream up to 40/50k tokens like this using 1 API call.

I also built in some fault tolerance and output recovery, by occasionally storing the output during streaming into KV (Durable Objects storage is also a possibility). If it fails midway and retries in my workflow, it attempts to restore the state and resume where it left off.

What I really like is that I can send real-time updates this way to consumers, since I prompted Gemini to occasionally send an [status: {...}] marker which I can parse in real-time, since outputting 40/50k tokens can time some time.

ItsWendell•4/23/25, 12:59 PM

https://discord.com/channels/595317990191398933/1040420029080018945/1364574169672450112

Awesome that the proxy limit has been increased beyond 100 seconds for AI Gateway, what is the new limit? Just so I can adjust my internal timeouts / retry logic!

ItsWendell•4/23/25, 10:08 PM

@Gabriel nice! Is that only for requests that are

workers <> ai gateway <> model provider

workers <> ai gateway <> model provider

? Or also

workers <> model providers

workers <> model providers

, another case that I'm hitting is that I'm creating context caches at Google Gemini APIs that can take quite a long time, with larger contexts, e.g. a lot of big files, this takes over 100 seconds and I can't use the API gateway for this as far as I know.

IItsWendell @Gabriel nice! Is that only for requests that are `workers <> ai gateway <> mod...

Kathy•4/24/25, 4:21 PM

if requests are not going through AI Gateway, you might need to increase this limit for your domain. Looks like it may be an enterprise only capability https://developers.cloudflare.com/fundamentals/reference/connection-limits/#between-cloudflare-and-origin-server

Cloudflare Docs

Connection limits · Cloudflare Fundamentals docs

When HTTP/HTTPS traffic is proxied through Cloudflare, there are often two established TCP connections: the first is between the requesting client to Cloudflare and the second is between Cloudflare and the origin server. Each connection has their own set of TCP and HTTP limits, which are documented below.

KKathy if requests are not going through AI Gateway, you might need to increase this li...

ItsWendell•4/24/25, 4:41 PM

Thanks for coming back to this! I was aware of the limits and the ability of getting an enterprise plan to upgrade these limits, but that would be for our business in an MVP / prototyping phase not feasable.

If context caching would be supported in AI Gateway, that would be great since than we could also use features like tracking costs of that (in google's case it returns usage, unsure if other providers offer similar methods).

IItsWendell Thanks for coming back to this! I was aware of the limits and the ability of get...

Kathy•4/24/25, 4:49 PM

explain more what you meant by your last sentence on context caching? initially i thought that was re: semantic caching, but seems you're talking about something different

ItsWendell•4/24/25, 5:02 PM

@Kathy | Browser Rendering PM Ah fair, it's related to "context caching" at Google / Vertex AI. For huge contexts and long repeated runs with the same context this can siginificantly speed up inference and cost savings. https://ai.google.dev/gemini-api/docs/caching?lang=rest.

The problem I'm having here is that Google is taking an increasingly long time to more context you give it to create this Context Cache. We often hit 2+ minutes for bigger context (20/30+ PDF files, some of them 100+ pages, which on their side I think are being transformed into images, and then into tokens).

Google AI for Developers

Context caching | Gemini API | Google AI for Developers

Learn how to use Context Caching in the Gemini API

Erisa•4/24/25, 9:45 PM

Is it expected or a bug that requesting workers ai openai-compatible api thru ai gateway (specifically with gemma3 in this case) would not log the request and only the response as shown in the screenshot?

IItsWendell @Kathy | Browser Rendering PM Ah fair, it's related to "context caching" at Goog...

Kathy•4/24/25, 11:49 PM

thanks for sharing so the ask here is for ai gateway to support tracking the costs of context caching through providers, including google

what does the response look like from google when using conext caching? Curious to see how it splits out input, output, and context caching because that is how we would track tokens to then calculate

EErisa Is it expected or a bug that requesting workers ai openai-compatible api thru ai...

Kathy•4/24/25, 11:49 PM

thisssss seems like a bug. could you share your account id + gateway name through DM please

KKathy thisssss seems like a bug. could you share your account id + gateway name throug...

mr.niko.la•4/26/25, 10:04 AM

Hi will you share whisper turbo sample code to test it. Could use a repo with examples.

xpawlik92x•4/26/25, 5:12 PM

Hey!

I'm currently checking AI Gateway for the company I work at — mainly to track token usage per our customer.

While testing GPT-4.1 with streaming enabled, I noticed that if a client cancels a request mid-stream, there’s no trace of that request in the AI Gateway logs.
Is this the intended behavior, or could this be a bug or a limitation of the current implementation?

Azuredush•5/1/25, 10:23 AM

When using Aistudio's OpenAI-compatible endpoint through AI Gateway, a lot of "��" characters appear when the model outputs long content. However, when I use the official endpoint directly, everything works fine. Why is this happening?

AAzuredush When using Aistudio's OpenAI-compatible endpoint through AI Gateway, a lot of "�...

Isaac McFadyen•5/1/25, 4:16 PM

That's probabably not long content but because it's a UTF-8 character that can't be rendered on the Cloudflare dashboard because it's missing in the font they use.

rob•5/2/25, 2:16 PM

import { createWorkersAI } from 'workers-ai-provider'; import { streamText } from 'ai'; ... const model = '@cf/meta/llama-3.1-70b-instruct'; const workersai = createWorkersAI({ binding: env.AI, gateway: { id: "ai-gateway-id ?" }, baseURL: 'https://gateway.ai.cloudflare.com/...?' }); ... const result = streamText({ model: workersai(model), prompt: finalPrompt, });

import Anthropic from "@anthropic-ai/sdk"; const anthropic = new Anthropic({ apiKey: "<api key>", baseURL: "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/grok", }); const msg = await anthropic.messages.create({ model: "grok-beta", max_tokens: 128, system: "You are Grok, a chatbot inspired by the Hitchhiker's Guide to the Galaxy.", messages: [ { role: "user", content: "What is the meaning of life, the universe, and everything?", }, ], }); console.log(msg);

Could you tell me how long it will take for Llama 4 to be integrated?

Similar Threads

Similar Threads

Similar Threads