Cloudflare Developers•10mo ago

https://developers.cloudflare.com/ai-gateway/guardrails/

IIsaac McFadyen https://developers.cloudflare.com/ai-gateway/guardrails/

龙

龙卷风•3/17/25, 5:26 PM

I see, thank you.

Rrob https://x.com/G4brym/status/1901413193970299149 did u see this ?

dom•3/20/25, 8:16 AM

I just took a peak. Not exactly what I need. With this solution I'm still required to define target models at app level.

ring0•3/20/25, 3:51 PM

Has anyone been successful using the openai python library to call AzureOpenAI through Cloudflare Gateway?

Rring0 Has anyone been successful using the openai python library to call AzureOpenAI ...

ring0•3/20/25, 7:00 PM

figured it out. in case anyone needs this, I was missing the default_query:

client = OpenAI(
default_query={"api-version": $AZURE_OPENAI_API_VERSION)},
base_url=$api_base,
api_key=$AZURE_OPENAI_API_KEY),
)

Victor•3/21/25, 9:22 PM

I'm using custom costs but it mentions

Custom costs will appear in the logs with an underline, making it easy to identify when custom pricing has been applied.

But that is no longer the case (I haven't used custom costs before so I don't know when it disappeared).

Cloudflare Docs

Custom costs · Cloudflare AI Gateway docs

Override default or public model costs on a per-request basis.

Ddom I have a product idea. An AI Router. Similar to AI Gateway but it would allow me...

Kathy•3/26/25, 4:27 AM

thanks for the ideas - we're working on an openai compatible endpoint that will help with #2 - saving you from refactoriing the code for different models/providers. will do some thinking on the routing

VVictor I'm using [custom costs](https://developers.cloudflare.com/ai-gateway/configurat...

Kathy•3/26/25, 4:36 AM

just tested and seems to be working. what did you try?

one thing i did notice is for decimals you need to put "0.2". It can't be ".2"

donp•3/31/25, 3:20 AM

Is adding support for Elevenlabs new Speech to Text model called Scribe on the roadmap?

https://developers.cloudflare.com/ai-gateway/providers/elevenlabs/

I tried replacing "text-to-speech" with "speech-to-text", but the endpoint doesn't exist.

Scribe docs - https://elevenlabs.io/docs/capabilities/speech-to-text

Cloudflare Docs

ElevenLabs · Cloudflare AI Gateway docs

ElevenLabs offers advanced text-to-speech services, enabling high-quality voice synthesis in multiple languages.

Speech to Text | ElevenLabs Documentation

Learn how to turn spoken audio into text with ElevenLabs.

smile•3/31/25, 1:16 PM

Hi Everyone. I'm trying to integrate the cloudflare non-realtime websocket api. But I faced the following issues. The WebSocket connection is established, but after 1 or 2 messages, the event listener stops triggering. Please guide me to resolve it.

smile•3/31/25, 1:17 PM

https://developers.cloudflare.com/ai-gateway/configuration/websockets-api/non-realtime-api/

Based on this doc I have created the sample.

Cloudflare Docs

Non-realtime WebSockets API · Cloudflare AI Gateway docs

The Non-realtime WebSockets API allows you to establish persistent connections for AI requests without requiring repeated handshakes. This approach is ideal for applications that do not require real-time interactions but still benefit from reduced latency and continuous communication.

smile•3/31/25, 1:20 PM

Does this API work only with a paid account? First, I want to test a working sample for both real-time and non-real-time WebSocket APIs to evaluate their performance and reduce latency for chat completion.

Milo•3/31/25, 9:02 PM

Hi guys, I got banned from AI Bot from Cloudfare said my provider, he had the AI Bot detection on and he said he wont whitelist me its against protection. So how do I approach Cloudfare to fix this?

MMilo Hi guys, I got banned from AI Bot from Cloudfare said my provider, he had the AI...

Isaac McFadyenOP•3/31/25, 9:22 PM

?crossposting AI BOT banned me by mistake and my provider tells me I need to handle it with Cloudfare?

SuperHelpflare•3/31/25, 9:22 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

IIsaac McFadyen ?crossposting https://discord.com/channels/595317990191398933/135637501997809670...

Milo•3/31/25, 9:23 PM

Sorry, I was not able to find it so fast, there are many channels here

glncy•4/1/25, 7:48 AM

Hi to all. I am trying to access AI Gateway and Workers AI locally but it returning a same error. I've been stuck since yesterday about this

Thank you very much

Additional context: I tried deploying a sample worker ai on cloudflare workers. it is working thought. i am not sure what i am doing wrong

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 7010,
      "message": "Service unavailable"
    }
  ],
  "messages": [
    
  ]
}

{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 7010,
      "message": "Service unavailable"
    }
  ],
  "messages": [
    
  ]
}

eleng•4/1/25, 9:16 AM

Hi! I’d like to use AI Gateway as a proxy for my Replicate model.

I see Replicate listed as a supported provider, and the docs mention it's “supported out of the box.” However, I don’t see any configuration options to route requests to Replicate — and in the “Connect to your applications with API Endpoints” popup, Replicate isn’t listed among the AI Platforms.

Is it currently possible to use AI Gateway as a proxy for Replicate? If so, how can I set that up?

smile•4/1/25, 1:20 PM

Hi anyone help me to resolve the issue in AI Gateway (Non streaming websocket api). The WebSocket connection is established, but after 1 or 2 messages, the event listener stops triggering. I have provided the code. Please check it out and guide me to resolve the issue.

message.txt4.26KB

Victor•4/1/25, 10:19 PM

Getting

2009 Unauthorized

2009 Unauthorized

consistently on every request going to any authenticated gateway (any provider/model). I rolled a new ai gateway token and azure ai tokens and manually tried them, but they're failing too. Last known good request was April 1, 2025

Victor•4/1/25, 10:20 PM

(testing with azure openai specifically - just the random one I picked) Without changing anything of the request other than the url, I swapped it to hit openai directly (without gateway) and it went through

200 OK

200 OK

Isaac McFadyenOP•4/1/25, 10:30 PM

^ @michelle unsure if this is related to the other issue but might be worth a look at?

VVictor Getting `2009 Unauthorized` consistently on every request going to any authentic...

Isaac McFadyenOP•4/1/25, 10:32 PM

Mind sending your account ID?

IIsaac McFadyen Mind sending your account ID?

Victor•4/1/25, 10:32 PM

e9b82b39aa91575a2643529d24a1bf02

e9b82b39aa91575a2643529d24a1bf02

Isaac McFadyenOP•4/1/25, 10:45 PM

For anyone following this issue, it's been escalated internally.

Eeleng Hi! I’d like to use AI Gateway as a proxy for my Replicate model. I see Replica...

Kathy•4/1/25, 10:49 PM

https://developers.cloudflare.com/ai-gateway/providers/replicate/

Cloudflare Docs

Replicate · Cloudflare AI Gateway docs

Replicate runs and fine tunes open-source models.

VVictor Getting `2009 Unauthorized` consistently on every request going to any authentic...

Kathy•4/2/25, 12:08 AM

thanks for bringing to our attention

. fixed and should be working now

Ddom I just took a peak. Not exactly what I need. With this solution I'm still requir...

rob•4/5/25, 5:54 PM

well if you wanted to talk more abut it lmk

Vladimir_dev•4/5/25, 8:04 PM

Could you tell me how long it will take for Llama 4 to be integrated?

akshay•4/6/25, 2:37 PM

Hey folks!

I'm currently exploring how to build with OpenAI's real-time API and Cloudflare. Most examples I've come across are based on WebSockets, but I'm considering using WebRTC instead due to potential performance benefits.

Does anyone have references or examples of using WebRTC in this context? Would love to learn from any prior work or suggestions. Thanks in advance!

VVladimir_dev Could you tell me how long it will take for Llama 4 to be integrated?

Kathy•4/7/25, 11:02 PM

Are you referring to Workers AI? If so, that was added yesterday https://blog.cloudflare.com/meta-llama-4-is-now-available-on-workers-ai/
there's a different channel for Workers AI workers-ai

The Cloudflare Blog

Meta Llama 4 now available on Workers AI

Llama 4 Scout 17B Instruct is now available on Workers AI: use this multimodal, Mixture of Experts AI model on Cloudflare's serverless AI platform to build next-gen AI applications. Build products that can understand images & text, accept large context, with high performance on model quality and inference latency.

Aakshay Hey folks! 👋 I'm currently exploring how to build with OpenAI's real-time API a...

Kathy•4/7/25, 11:03 PM

webRTC not yet supported in AI Gateway but will in the future. In the meantime, can check out Calls https://developers.cloudflare.com/calls/

Cloudflare Docs

Overview · Cloudflare Calls docs

Cloudflare Calls is infrastructure for real-time audio/video/data applications. It allows you to build real-time apps without worrying about scaling or regions. It can act as a selective forwarding unit (WebRTC SFU), as a fanout delivery system for broadcasting (WebRTC CDN) or anything in between.

smile•4/8/25, 4:03 AM

Hi all, I've been trying to work with the Cloudflare WebSocket API, but it isn't functioning as expected. I've reached out to the Cloudflare community and Discord for support, but unfortunately, I haven’t received any response so far.

Could someone please help me identify whether the issue lies with the Cloudflare API itself or if there's something wrong in my implementation?

Jaka•4/8/25, 5:08 AM

i have AutoRag running from yesterday. nothing seems to happen. Things appear blocked. Need help.

Bryan Chen•4/8/25, 8:40 AM

Any plan to improve the logs viewer? Currently the requests and responses are just rendered as JSON, which makes it impossible to read (due to

\n

\n

s). I am really considering build a 3rd party viewer for it.

Original message was deleted

smile•4/9/25, 8:07 AM

Hi, Did you use Webscoker API in cloudflare?

Radek D.•4/9/25, 4:46 PM

hi all I got stuck on this I want to call from AI Agent to AI gateway via binding env.AI but can I call public providers OpenAI/Anthropic etc or only CF models ? https://developers.cloudflare.com/agents/api-reference/using-ai-models/#model-routing

Cloudflare Docs

Using AI Models · Cloudflare Agents docs

Agents can communicate with AI models hosted on any provider, including:

Radek D.•4/9/25, 4:47 PM

if I want to call public providers I need to call AI Gatewat directly via HTTP or wss or can use env.AI binding ? apreciate some help

Radek D.•4/9/25, 5:10 PM

https://developers.cloudflare.com/ai-gateway/integrations/worker-binding-methods/ found it !

Cloudflare Docs

AI Gateway Binding Methods · Cloudflare AI Gateway docs

This guide provides an overview of how to use the latest Cloudflare Workers AI Gateway binding methods. You will learn how to set up an AI Gateway binding, access new methods, and integrate them into your Workers.

AAppSail Hi, Why did this happen?

Kathy•4/11/25, 5:45 PM

this is an issue with OpenRouter returning an error but with a 200 http code

JJaka i have AutoRag running from yesterday. nothing seems to happen. Things appear bl...

Kathy•4/11/25, 5:45 PM

ai-search

AAppSail Hi, Why did this happen?

Kathy•4/11/25, 5:46 PM

Could you send me your account info please? we can make a fix for this

KKathy this is an issue with OpenRouter returning an error but with a 200 http code

dave•4/14/25, 11:02 AM

Sadly for steaming requests, this is always gonna be a limitation.

It sucks HTTP doesn’t allow amending/setting headers after the response body.

Ddave Sadly for steaming requests, this is always gonna be a limitation. 😦 It sucks H...

Isaac McFadyenOP•4/14/25, 3:16 PM

Except that here it wasn't streamed.

Isaac McFadyenOP•4/14/25, 3:16 PM

Presumably they have enough information about the response by the time they send the HTTP status that they could change it if there's an error

m4rs0n•4/15/25, 4:12 AM

Subject: Grok API Location Detection Issue When Using ai-gateway from Mainland China

Description:

Hello Cloudflare team,

I am a developer based in Mainland China. I am currently using ai-gateway to call the Grok API. I have noticed that when I make requests directly (without a VPN) from my local developing Mac, Grok is able to detect my actual location and refuse my request. However, when I use a VPN to make the same Grok API calls, this location detection issue does not occur.

Could you please help me understand why Grok is able to recognize my real location when I am not using a VPN, and whether there are any configuration options or best practices to prevent this? Is this expected behavior for users in Mainland China, or is there something specific to ai-gateway or my setup that I should be aware of?

Thank you for your assistance!

cosbgn•4/16/25, 12:12 PM

I have an app where users add their own OpenAI api key. So the Auth header will be different from user to user.

Can I still use

ai-gateway

ai-gateway

or cache will get mixed up?

rufi•4/16/25, 1:41 PM

how can i integrate the ai-gateway with createWorkersAI (workers-ai-provider)?

import { createWorkersAI } from 'workers-ai-provider';
import { streamText } from 'ai';
...
const model = '@cf/meta/llama-3.1-70b-instruct';
    const workersai = createWorkersAI({ binding: env.AI, 
        gateway: {
          id: "ai-gateway-id ?"
        },
        baseURL: 'https://gateway.ai.cloudflare.com/...?'
      });
      ...
    const result = streamText({
      model: workersai(model),
      prompt: finalPrompt,
    });

import { createWorkersAI } from 'workers-ai-provider';
import { streamText } from 'ai';
...
const model = '@cf/meta/llama-3.1-70b-instruct';
    const workersai = createWorkersAI({ binding: env.AI, 
        gateway: {
          id: "ai-gateway-id ?"
        },
        baseURL: 'https://gateway.ai.cloudflare.com/...?'
      });
      ...
    const result = streamText({
      model: workersai(model),
      prompt: finalPrompt,
    });

luiseok•4/17/25, 5:52 AM

AI Gateway is currently returning a 500 internal server error. Am I being affected by API performance degradation?

abuiles•4/19/25, 1:38 AM

Minor nit, but it seems like the logs filter UI is assuming strings in the metadata.

The example in the docs shows this

--header 'cf-aig-metadata: {"team": "AI", "user": 12345, "test":true}' \

--header 'cf-aig-metadata: {"team": "AI", "user": 12345, "test":true}' \

But then when you try to filter in the UI it won't work

filters: [{"key":"metadata.value","operator":"eq","value":["12345"]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

filters: [{"key":"metadata.value","operator":"eq","value":["12345"]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

I changed the request manually to a a number and it worked

filters: [{"key":"metadata.value","operator":"eq","value":[12345]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

filters: [{"key":"metadata.value","operator":"eq","value":[12345]},{"key":"metadata.key","operator":"eq","value":["user"]},{"key":"created_at","operator":"gt","value":["2025-04-18T01:21:00.000Z"]}]

Is this expected?

import { createWorkersAI } from 'workers-ai-provider'; import { streamText } from 'ai'; ... const model = '@cf/meta/llama-3.1-70b-instruct'; const workersai = createWorkersAI({ binding: env.AI, gateway: { id: "ai-gateway-id ?" }, baseURL: 'https://gateway.ai.cloudflare.com/...?' }); ... const result = streamText({ model: workersai(model), prompt: finalPrompt, });

https://developers.cloudflare.com/ai-gateway/guardrails/

Similar Threads