Is output token limit to 256 now? - Cloudflare Developers

© 2026 Hedgehog Software, LLC

More

Communities Docs About Terms Privacy

tkisme

Ttkisme Is output token limit to 256 now?

James

J

James•2/26/25, 2:54 PM

defaults are enforced now it seems, which hopefully will be more loudly communicated soon, as it can cause breaking changes for people's apps. cc @michelle
You should be able to increase max_tokensmax_tokens to tweak this.

Johnny Bernhard

J

Johnny Bernhard•2/27/25, 3:10 PM

Is there a way to limit the new guardrail to specific operations in a gateway?
For our use-case it would be enough to enable the guardrails on the embedding step before RAG. We don't need to run them afterwards.

fatwang2

Ffatwang2 Too many providers for you to add,but most of them are OpenAI-Compatible-Endpoin...

thomash

T

thomash•2/28/25, 4:53 PM

i ended up using the portkey ai gateway.

https://github.com/Portkey-AI/gateway

the good thing is you can deploy it straight to cloudflare workers and it allows using custom openai endpoints

GitHub

GitHub - Portkey-AI/gateway: A blazing fast AI Gateway with integra...

A blazing fast AI Gateway with integrated guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. - Portkey-AI/gateway

thomash

Tthomash i ended up using the portkey ai gateway. https://github.com/Portkey-AI/gateway...

fatwang2

F

fatwang2•2/28/25, 7:09 PM

I am on litellm now..

fatwang2

F

fatwang2•2/28/25, 7:09 PM

similar to portkey

fatwang2

Ffatwang2 I am on litellm now..

thomash

T

thomash•2/28/25, 7:33 PM

i looked at both and portkey seemed more production ready and easy to deploy serverless to cloudflare. but im sure litellm is also fine. it has many users

thomash

Tthomash i looked at both and portkey seemed more production ready and easy to deploy ser...

fatwang2

F

fatwang2•3/1/25, 6:13 AM

how about the latency

fatwang2

Ffatwang2 how about the latency

thomash

T

thomash•3/1/25, 6:44 PM

portkey is much faster but usually latency is constrained by the LLM not the gateway

numberstorm

N

numberstorm•3/2/25, 9:47 AM

hey, how do i give access to my domain to a friend

Johnny Bernhard

JJohnny Bernhard Is there a way to limit the new guardrail to specific operations in a gateway? F...

Kathy

K

Kathy•3/3/25, 5:37 PM

hm so you would not want to run on the prompts where you're querying RAG? (this "knowledge query" part of RAG https://developers.cloudflare.com/reference-architecture/diagrams/ai/ai-rag/#knowledge-queries

For embeddings, we only evaluate the prompts https://developers.cloudflare.com/ai-gateway/guardrails/supported-model-types/

Cloudflare Docs

Supported model types · Cloudflare AI Gateway docs

AI Gateway's Guardrails detects the type of AI model being used and applies safety checks accordingly:

Cloudflare Docs

Retrieval Augmented Generation (RAG) · Cloudflare Reference Archite...

Retrieval-Augmented Generation (RAG) is an innovative approach in natural language processing that integrates retrieval mechanisms with generative models to enhance text generation.

Bryan Chen

BBryan Chen for custom costs, does it able to take cache hit/miss into consideration? otherw...

Kathy

K

Kathy•3/3/25, 6:37 PM

if it's a cache HIT the cost will be 0. Even if you have passed a custom cost

Kathy

K

Kathy•3/3/25, 6:57 PM

also, ICYMI! question about guardrails above is because we launched guardrails last week

https://blog.cloudflare.com/guardrails-in-ai-gateway/

please try and let us know what you think~

The Cloudflare Blog

Keep AI interactions secure and risk-free with Guardrails in AI Gat...

Deploy AI safely with built-in Guardrails in AI Gateway. Flag and block harmful or inappropriate content, protect personal data, and ensure compliance in real-time — keeping AI interactions secure and risk-free.

boris (^_^)

B

boris (^_^)•3/3/25, 7:25 PM

Is there a way to use AI gateway for models that are not supported? I’m thinking of image models API where it would be cool to have the request go through the AI gateway to log requests/costs/analytics…
In particular I’m thinking of something kind of universal where we just prepend the full endpoint with our gateway url and inject correct headers.

Kathy

KKathy if it's a cache HIT the cost will be 0. Even if you have passed a custom cost

Bryan Chen

B

Bryan Chen•3/3/25, 10:24 PM

by cache hit, I mean the one on anthropic, not the one on CF https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

Anthropic

Prompt caching - Anthropic

Bryan Chen

B

Bryan Chen•3/4/25, 2:42 AM

it seems like it is not possible to use AI binding to streaming request?

x3gakmafia01tm

X

x3gakmafia01tm•3/4/25, 2:53 AM

سلام

Bryan Chen

B

Bryan Chen•3/4/25, 6:27 AM

I am trying AI Gateway with WebSocket and what is the pp field in the stream response?

  "streamed_data": [
    {
      "response": "As",
      "p": "abcde"
    },
    {
      "response": " an",
      "p": "ab"
    },
    {
      "response": " artificial",
      "p": "abcdefghijklmnopqrs"
    },
    {
      "response": " intelligence",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcdefghi"
    },
    {
      "response": " language",
      "p": "abcdefghi"
    },
    {
      "response": " model",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": ",",
      "p": "abcdefghijklmn"
    },
    {
      "response": " I",
      "p": "abcd"
    },
    {
      "response": " don",
      "p": "abcdefghijklmnopqrstuvwxyz012"
    },
    {
      "response": "'t",
      "p": "abcdefghijklmn"
    },
    {
      "response": " have",
      "p": "abcdefgh"
    },
    {
      "response": " a",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcde"
    },
    {
      "response": " specific",
      "p": "abcdefg"
    },
    {
      "response": " name",
      "p": "abcd"
    },
    {
      "response": ".",
      "p": "abcd"
    },
    {
      "response": "",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": "",
      "usage": {
        "prompt_tokens": 24,
        "completion_tokens": 17,
        "total_tokens": 41
      }
    }
  ]
}

  "streamed_data": [
    {
      "response": "As",
      "p": "abcde"
    },
    {
      "response": " an",
      "p": "ab"
    },
    {
      "response": " artificial",
      "p": "abcdefghijklmnopqrs"
    },
    {
      "response": " intelligence",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcdefghi"
    },
    {
      "response": " language",
      "p": "abcdefghi"
    },
    {
      "response": " model",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": ",",
      "p": "abcdefghijklmn"
    },
    {
      "response": " I",
      "p": "abcd"
    },
    {
      "response": " don",
      "p": "abcdefghijklmnopqrstuvwxyz012"
    },
    {
      "response": "'t",
      "p": "abcdefghijklmn"
    },
    {
      "response": " have",
      "p": "abcdefgh"
    },
    {
      "response": " a",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcde"
    },
    {
      "response": " specific",
      "p": "abcdefg"
    },
    {
      "response": " name",
      "p": "abcd"
    },
    {
      "response": ".",
      "p": "abcd"
    },
    {
      "response": "",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": "",
      "usage": {
        "prompt_tokens": 24,
        "completion_tokens": 17,
        "total_tokens": 41
      }
    }
  ]
}

Bryan Chen

B

Bryan Chen•3/4/25, 6:45 AM

websocket steaming mode doesn't seem to be working with deepseek. I received no response via websocket. however I can see response in the dashboard logs

Kathy

KKathy hm so you would not want to run on the prompts where you're querying RAG? (this ...

Johnny Bernhard

J

Johnny Bernhard•3/4/25, 3:24 PM

Yes. It's a super simple flow of:
Embed query and retrieve from Vector Store
Build a big prompt with query + RAG results
Send that to LLM

There is no need to double guardrail it. The prompt is already checked in the embedding step.
And the big query is build from safe-bits (guard before entrying into DB + guarded user query).

So the second guard is just extra latency.

Bryan Chen

BBryan Chen I am trying AI Gateway with WebSocket and what is the `p` field in the stream re...

Isaac McFadyen

I

Isaac McFadyen•3/4/25, 3:56 PM

https://blog.cloudflare.com/sv-se/ai-side-channel-attack-mitigated/

The Cloudflare Blog

Mitigating a token-length side-channel attack in our AI products

The Workers AI and AI Gateway team recently collaborated closely with security researchers at Ben Gurion University regarding a report submitted through our Public Bug Bounty program. Through this process, we discovered and fully patched a vulnerability affecting all LLM providers. Here’s the story

Peps

P

Peps•3/5/25, 3:22 AM

Does anyone know if disabling and re-enabling cache on a gateway clears it's previous cache?

Nisar446

N

Nisar446•3/5/25, 7:26 PM

Hi

ari

A

ari•3/5/25, 8:05 PM

We just implemented AI Gateway using the Universal endpoint, with both OpenAI and Anthropic models in the list. Is there a way to test that the fallback is working (short of waiting for the next OpenAI service outage)? We've tried providing an invalid API key for the primary service from a local environment, but it seems like this would return a different error. What are the recommended ways to test different failure modes?

ari

A

ari•3/5/25, 8:07 PM

Also wondering what the "predetermined request timeout" is. The documentation on this is just a broken link (https://developers.cloudflare.com/ai-gateway/configuration/fallbacks/#request-timeouts:~:text=predetermined%20request%20timeouts)

ari

A

ari•3/6/25, 5:19 PM

One more issue: Turning on Guardrails seems to break streaming (at least when OpenAI is the default model.)

Johnny Bernhard

JJohnny Bernhard Yes. It's a super simple flow of: Embed query and retrieve from Vector Store Bui...

Kathy

K

Kathy•3/7/25, 7:28 PM

got it, we currently don't have per request control to disable guardrails. so thanks for sharing your use - something for us to think about

as for how to help right now, since guardrails is at the gateway level, you could use two gateways - one for embedding with guardrails on. one for queries wiht guardrails off. sorry not the cleanest

ari

Aari One more issue: Turning on Guardrails seems to break streaming (at least when Op...

Kathy

K

Kathy•3/7/25, 7:29 PM

you're right, good point - will add to docs. currently doesn't work for streaming

ari

Aari We just implemented AI Gateway using the Universal endpoint, with both OpenAI an...

Kathy

K

Kathy•3/7/25, 7:29 PM

the way you tried of providing an invalid key to force the fallback does work

 [
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN-INVALID",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+1=?"
                }
            ]
        }
    },
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+10=?"
                }
            ]
        }
    }
]

 [
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN-INVALID",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+1=?"
                }
            ]
        }
    },
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+10=?"
                }
            ]
        }
    }
]

`

ari

Aari Also wondering what the "predetermined request timeout" is. The documentation on...

Kathy

K

Kathy•3/7/25, 7:30 PM

thanks for pointing that out. fixed

Kathy

KKathy got it, we currently don't have per request control to disable guardrails. so th...

Johnny Bernhard

J

Johnny Bernhard•3/8/25, 10:56 AM

Yeah, two Gateways would work, but a bit annoying to set up 2 more gateways (prod and dev) and share the config around.
But definitely understand this not being too important to change right now.

Kathy

KKathy not at the moment. our rate limits right now are by number of requests through a...

Super Morris

S

Super Morris•3/8/25, 4:51 PM

please inplement ratelimit per user/metadata.... that would be awsome!

Super Morris

S

Super Morris•3/8/25, 4:52 PM

that would help everybody building apps with ai that want to rate limit their users usage out of the box.

nclevenger

N

nclevenger•3/9/25, 9:59 AM

When using openrouter, the duration values are way lower than reality ... many of these requests are taking multiple seconds (or even 10s of seconds) ... and even the fastest ones are coming back in mid-to-high hundreds of milliseconds while AI Gateway isn't reporting anything over 225ms ...

The other [minor but annoying] issue with AI Gateway + OpenRouter is that the model names are cut off very short, and given the {model}:{keyword}{model}:{keyword} syntax used by open router, the end of the model name is really critical to see when streaming or scanning the last

Shawn

S

Shawn•3/11/25, 2:25 AM

Can anyone help with this issue or know another site to go to?

Shawn

SShawn Can anyone help with this issue or know another site to go to?

Isaac McFadyen

I

Isaac McFadyen•3/11/25, 2:02 PM

?crossposting

SuperHelpflare

S

SuperHelpflare•3/11/25, 2:02 PM

Please do not post your question in multiple channels/post it multiple times per the rules at #

welcome-and-rules. It creates confusion for people trying to help you and doesn't get your issue or question solved any faster.

Peps

PPeps Does anyone know if disabling and re-enabling cache on a gateway clears it's pre...

Kathy

K

Kathy•3/11/25, 9:31 PM

disabling and re-enabling will not clear it's previous cache, sorry

Right now the only option is to wait for the ttl to expire --> we need to add the ability to purge cache still

Super Morris

SSuper Morris please inplement ratelimit per user/metadata.... that would be awsome!

Kathy

K

Kathy•3/11/25, 9:34 PM

thanks for the feedback. helps with prioritization

nclevenger

Nnclevenger When using openrouter, the duration values are way lower than reality ... many o...

Kathy

K

Kathy•3/11/25, 9:35 PM

hm! might be a bug - can you dm me your account id please?

Kathy

KKathy hm! might be a bug - can you dm me your account id please?

nclevenger

N

nclevenger•3/12/25, 5:53 AM

just dm'd, thanks

Isaac McFadyen

IIsaac McFadyen https://blog.cloudflare.com/sv-se/ai-side-channel-attack-mitigated/

RobbBoss

R

RobbBoss•3/14/25, 8:50 PM

This is super interesting. Thank you for sharing that. From this, and based on the JSON output that was shared... well, I just devised another possible attack which might reveal the encryption keys themselves. It's possible this mitigation technique might create a compromise in its current implementation. Given that discussing it here would be irresponsible, who should I contact, please?

RobbBoss

RRobbBoss This is super interesting. Thank you for sharing that. From this, and based on t...

Isaac McFadyen

I

Isaac McFadyen•3/14/25, 8:54 PM

If you believe you've found a vulnerability you should report it through HackerOne.

Isaac McFadyen

I

Isaac McFadyen•3/14/25, 8:54 PM

?hackerone

SuperHelpflare

S

SuperHelpflare•3/14/25, 8:55 PM

This discord is not for for reporting security reports. Cloudflare has a Responsible Disclosure program run through HackerOne where you can privately file reports, which will be reviewed by the appropriate teams and for which you may be eligible for bug bounties: https://hackerone.com/cloudflare

HackerOne

Cloudflare Public Bug Bounty - Bug Bounty Program | HackerOne

The Cloudflare Public Bug Bounty Bug Bounty Program enlists the help of the hacker community at HackerOne to make Cloudflare Public Bug Bounty more secure. HackerOne is the #1 hacker-powered security platform, helping organizations find and fix critical vulnerabilities before they can be criminally exploited.

RobbBoss

R

RobbBoss•3/14/25, 8:56 PM

Thanks, I juuust saw that at the end of the article. haha Thank you for responding, though, truly.

I am likely to write about it, but not likely to go through the trouble of proving it out, which seems necessary to report it there. Well, thank you for that insight, regardless!

RobbBoss

RRobbBoss Thanks, I juuust saw that at the end of the article. haha Thank you for respondi...

Isaac McFadyen

I

Isaac McFadyen•3/14/25, 10:01 PM

If you are going to publicly write about it without reporting it then that would be a security issue. If you don't end up reporting it then you should not publicly write about it.

RobbBoss

R

RobbBoss•3/14/25, 10:03 PM

No, if I write about it I'll probably share it with researcher friends. I'm not going to randomly publicly disclose security issues.

RobbBoss

R

RobbBoss•3/14/25, 10:03 PM

Thank you for the due diligence in stating that.

I also have no interest in joining HackerOne, tbh. But I respect that there's a community of people that find bugs and there are bounties and all this. I have no interest in bounties at all. I would take this to #off-topic but I gather that this is probably the end of this thread.

RobbBoss

RRobbBoss Thank you for the due diligence in stating that. 🙂 I also have no interest in ...

Isaac McFadyen

I

Isaac McFadyen•3/14/25, 10:26 PM

It's less about bounties (although that's part of it) and more about responsible disclosure. By telling others that aren't Cloudflare, you risk them telling others themselves and eventually telling someone who is in a position to and wants to actively exploit it.

I would heavily encourage you to share through HackerOne, but if you don't want to I'd avoid sharing with anyone else regardless of whether they are friends or not.

dom

D

dom•3/15/25, 10:43 AM

I have a product idea. An AI Router. Similar to AI Gateway but it would allow me to define custom routes where I can define the target llm model and settings like temp, max_tokens etc. I would then route each app llm call to the custome AI Router endpoint. I would send only the prompts (system(optional) and user prompts). All other settings, like which llm to use, requst settings would be defined at AI router level. With one setting I can turn on AI Gateway on the router.

This would allow me to quickly adapt to the fast paced landscape of llm models. I can quickly reconfigure app's functionality to use different llm's and even run tests to quickly to validate outputs.

I'm wasting a lot of time updating and refactoring app code to stay up to date with llm models. Each time I need to:

find the llm api functionality. Try to remember what it does and understand it
refactor the code to use different model because i found a better one, cheaper one or the one im using will no longer available.
test in staging, test in production

I think there is a huge opportunity here for a cloud provider to own this pipeline. A great lock in for the customer as well. Please pass the idea to your product team to further explore. Thanks!

  "streamed_data": [
    {
      "response": "As",
      "p": "abcde"
    },
    {
      "response": " an",
      "p": "ab"
    },
    {
      "response": " artificial",
      "p": "abcdefghijklmnopqrs"
    },
    {
      "response": " intelligence",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcdefghi"
    },
    {
      "response": " language",
      "p": "abcdefghi"
    },
    {
      "response": " model",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": ",",
      "p": "abcdefghijklmn"
    },
    {
      "response": " I",
      "p": "abcd"
    },
    {
      "response": " don",
      "p": "abcdefghijklmnopqrstuvwxyz012"
    },
    {
      "response": "'t",
      "p": "abcdefghijklmn"
    },
    {
      "response": " have",
      "p": "abcdefgh"
    },
    {
      "response": " a",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcde"
    },
    {
      "response": " specific",
      "p": "abcdefg"
    },
    {
      "response": " name",
      "p": "abcd"
    },
    {
      "response": ".",
      "p": "abcd"
    },
    {
      "response": "",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": "",
      "usage": {
        "prompt_tokens": 24,
        "completion_tokens": 17,
        "total_tokens": 41
      }
    }
  ]
}

  "streamed_data": [
    {
      "response": "As",
      "p": "abcde"
    },
    {
      "response": " an",
      "p": "ab"
    },
    {
      "response": " artificial",
      "p": "abcdefghijklmnopqrs"
    },
    {
      "response": " intelligence",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcdefghi"
    },
    {
      "response": " language",
      "p": "abcdefghi"
    },
    {
      "response": " model",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": ",",
      "p": "abcdefghijklmn"
    },
    {
      "response": " I",
      "p": "abcd"
    },
    {
      "response": " don",
      "p": "abcdefghijklmnopqrstuvwxyz012"
    },
    {
      "response": "'t",
      "p": "abcdefghijklmn"
    },
    {
      "response": " have",
      "p": "abcdefgh"
    },
    {
      "response": " a",
      "p": "abcdefghijklmnopqrstuvwxyz0123456789abcde"
    },
    {
      "response": " specific",
      "p": "abcdefg"
    },
    {
      "response": " name",
      "p": "abcd"
    },
    {
      "response": ".",
      "p": "abcd"
    },
    {
      "response": "",
      "p": "abcdefghijklmnopqr"
    },
    {
      "response": "",
      "usage": {
        "prompt_tokens": 24,
        "completion_tokens": 17,
        "total_tokens": 41
      }
    }
  ]
}

 [
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN-INVALID",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+1=?"
                }
            ]
        }
    },
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+10=?"
                }
            ]
        }
    }
]

 [
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN-INVALID",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+1=?"
                }
            ]
        }
    },
    {
        "provider": "workers-ai",
        "endpoint": "@cf/meta/llama-2-7b-chat-int8",
        "headers": {
            "Authorization": "Bearer CF_TOKEN",
            "Content-Type": "application/json"
        },
        "config":{
            "skipCache": true
        },
        "query": {
            "messages": [
                {
                    "role": "system",
                    "content": "You are a friendly assistant"
                },
                {
                    "role": "user",
                    "content": "10+10=?"
                }
            ]
        }
    }
]