Next.js/Vercel AI/Streaming Issue

I want to make cloudflare aware of this issue. It seems using Cloudflare with Vercel's AI package (on Vercel hosting) if the proxy feature is on for Cloudflare steaming responses no longer works. https://github.com/vercel/ai/issues/633

GitHub

Stream Response does not work properly with Cloudflare DNS settings...

Description Issue In our production environment, the Stream Response doesn't seem to work as expected. Instead of streaming the response, it loads and immediately returns the full response. Int...

someguyOP•10/17/23, 12:30 PM

Just pinging this again, it seems Vercel is indicating it's on Cloudflare's end

joulev•10/18/23, 8:15 AM

same issue here, hope someone looks into this soon

ack•10/18/23, 5:20 PM

Hey @someguy can you walk me through your setup and where this is failing please? Want to confirm what Vercel is saying

someguyOP•10/18/23, 5:34 PM

Using vercel preview domain (on their subdomain), streaming from OpenAI works (using Vercel/Vercel's AI package with their streaming commands)

Using the live site proxy'd behind CF using their IP in an A record (76.76.21.21) the stream appears to just return as one massive chunk, so functionally still works but streaming just seems to be off.

Oddly enough I have a second project that uses Vercel on a subdomain which uses a cname instead (using CF proxy also) and streaming works. So I wonder if it's an issue between A record/Cname

someguyOP•10/18/23, 5:34 PM

let me know if this helps!

someguyOP•10/18/23, 5:39 PM

Just tested that theory and it doesn't seem to matter if it's an A record/Cname, must be different project details

someguyOP•10/18/23, 5:40 PM

If you want a test account feel free to DM me, I turned the proxy off and can confirm it works with it off, then stops working with it on

someguyOP•10/18/23, 5:42 PM

weirdly enough they both projects use the same vercel call and function

ack•10/18/23, 5:58 PM

interesting bug.... the CNAME website to the A rec website though?

someguyOP•10/18/23, 6:09 PM

Ignore the cname mention, it doesn't seem related

ack•10/18/23, 6:20 PM

ok so the only difference is that 1 is through the vercel preview domain and the other is a custom zone you have with vercel?

someguyOP•10/18/23, 6:21 PM

The main difference is turning the CF proxy option off/on is that causes the issue

someguyOP•10/18/23, 6:21 PM

so I can only guess it's CF causing the issue

ack•10/18/23, 6:24 PM

maybe... where is this CF proxy option you're interacting with? on vercel or is it when you change DNS to cloudflare?

someguyOP•10/18/23, 6:25 PM

This option

someguyOP•10/18/23, 6:25 PM

in Cloudflare

ack•10/18/23, 6:27 PM

what's the website? can DM too if you dont want to talk about it publicly.

someguyOP•10/18/23, 6:36 PM

https://rightblogger.com/

RightBlogger Blog

RightBlogger: AI-Powered Content Tools for Bloggers

RightBlogger is a collection of 50+ high-powered tools for bloggers to better research, create, optimize, and promote your content.

someguyOP•10/18/23, 6:36 PM

All functions that involve streaming require login (you can make a free account to submit one time per tool)

someguyOP•10/18/23, 6:37 PM

you'll see the response just is one large output rather than streaming

ack•10/18/23, 7:38 PM

thanks, will try to repro and see what's going on

someguyOP•10/18/23, 7:49 PM

thank you!

Jordy•10/24/23, 4:05 PM

Just here to say I have the exact same issue. Issue goes away when I disable the Cloudflare Proxy within Cloudflare. Any progress @ack ?

Ssomeguy Using vercel preview domain (on their subdomain), streaming from OpenAI works (u...

Jordy•10/24/23, 5:04 PM

I have the same issue when using a Vercel CNAME within my Cloudflare DNS settings. So we can probably rule that out. My project is also on a subdomain, like app.somewebsite.com

Jordy•10/25/23, 8:14 AM

@ack I have made a temporary reproduction here:

Cloudflare Proxy on: https://chat.jordyvandenaardweg.nl/
Cloudflare Proxy off: https://chat-proxy-off.jordyvandenaardweg.nl/
Not behind Cloudflare: https://nextjs-chat-cloudflare.vercel.app/

All URL's point to the same deployment.

Try it out with the prompt: what is ai?

That prompt gives a reasonable response length you can use to debug it

You'll notice the one behind the proxy is not streaming the responses, token by token, but just shows the response when it's done. So that takes a few seconds.

The one's NOT behind Cloudflare Proxy shows that streaming DOES work.

Jordy•10/25/23, 8:15 AM

Please do not share these URLs, as i'm using a OpenAI key and each requests will add to my usage there. But feel free to use it for debugging.

Jordy•10/25/23, 8:17 AM

The code is here, it used a Vercel AI Chatbot template, where I only removed the Auth/Github login and updated all the depedencies to their latest version:
https://github.com/jvandenaardweg/nextjs-chat-cloudflare

Jordy•10/25/23, 8:19 AM

This is the API route that handles the incoming OpenAI stream: https://github.com/jvandenaardweg/nextjs-chat-cloudflare/blob/main/app/api/chat/route.ts

Jordy•10/25/23, 8:42 AM

Just to clarify: I had streaming working on Vercel behind Cloudflare Proxy for months. But stopped working recently. According to messages I can find online from others with the same issue, it stopped working about 2 weeks ago.

Jordy•10/25/23, 8:54 AM

Messages I could find with the same issue:
https://github.com/vercel/ai/issues/633
https://community.cloudflare.com/t/buffering-of-responses-did-not-used-to-be-a-problem-now-is/567306
https://community.cloudflare.com/t/streaming-edge-function-hosted-in-vercel-is-buffered-by-cloudflare-proxy/567635
https://github.com/vercel/ai/issues/239#issuecomment-1756185750
https://twitter.com/baptisteArno/status/1710282829483888786
https://twitter.com/thetrungvu/status/1712861296549409017

Jordy•10/27/23, 11:26 AM

I've created a support ticket with the above reproduction info

Baptiste•10/27/23, 12:03 PM

Exactly the same issue on Typebot! It used to work but all of the sudden it started to Buffer the streaming response. (https://twitter.com/baptisteArno/status/1710282829483888786)

Baptiste•10/27/23, 12:04 PM

I'd say it's pretty critical as I have to disable Cloudflare proxy for now

Jordy•10/27/23, 12:05 PM

Yeah same, current work around is disabling Cloudflare Proxy completely. Also tried to disable almost everything using Page Rules, but that did not work. The only way to get it working is the off switch in the DNS settings for Cloudflare Proxy

Jordy•10/27/23, 12:05 PM

I'll keep you guys updated on the support ticket

Baptiste•10/27/23, 12:29 PM

Also opened a topic some time ago: general-helpStreaming is buffered when Cloudflare proxy is on?

ack•10/27/23, 7:28 PM

Thank you, it took me a while but I was able to repro. Can people see if they are still experiencing the issue now that a change has gone out?

Jordy•10/27/23, 9:45 PM

Got a response back, they fixed it! See behaviour on reproduction: https://chat.jordyvandenaardweg.nl/ , the response now comes streaming in again. Also enabled Cloudflare Proxy on my production app, and all works again

Jordy•10/27/23, 9:45 PM

I'll be removing the reproduction demo's later.

Jordy•10/27/23, 9:45 PM

Thanks all!

ack•10/27/23, 9:45 PM

woot

Jordy•10/27/23, 9:47 PM

Aack Thank you, it took me a while but I was able to repro. Can people see if they ar...

Jordy•10/27/23, 9:54 PM

Thanks @ack

joulev•10/28/23, 2:16 AM

everything works now, thanks a lot to everyone involved!

someguyOP•10/28/23, 3:09 AM

Working for me woooo

FooQoo•10/28/23, 6:34 AM

Thanks!

swiecki•11/1/23, 4:36 AM

Hi, I'm not using vercel AI, but these recent messages came up when I searched for streaming response issues. I'm unable to get a cloudflare worker to stream any responses using either readable stream or transform stream. Instead it's being buffered and sent as a single chunk once the response is complete. Are there specific configuration requirements? I've tried setting transfer-encoding header to chunked and said no-cache but it's still an issue. Is there a way to upgrade my workers or something so they can respond with streaming responses? Thanks

Jordy•11/1/23, 7:57 AM

@swiecki i suggest to create a new topic here #general-help so others can help you with your specific issue

Sswiecki Hi, I'm not using vercel AI, but these recent messages came up when I searched f...

anurag•3/1/24, 11:42 AM

@swiecki were you able to find a solution for this? I'm losing my mind, spend hours thinking it's an issue with my implementation

Aanurag @swiecki were you able to find a solution for this? I'm losing my mind, spend ho...

swiecki•3/8/24, 7:17 AM

@anurag here's what worked for me

return new Response(readable, {
      headers: {
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "GET, POST",
        "Access-Control-Allow-Headers": "accept, content-type",
        "Content-Type": "text/event-stream",
      "Connection": "keep-alive",
      "Cache-Control": "no-cache",
      "X-Accel-Buffering": "no",
      "Content-Encoding": "none",
      }
    });

return new Response(readable, {
      headers: {
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Methods": "GET, POST",
        "Access-Control-Allow-Headers": "accept, content-type",
        "Content-Type": "text/event-stream",
      "Connection": "keep-alive",
      "Cache-Control": "no-cache",
      "X-Accel-Buffering": "no",
      "Content-Encoding": "none",
      }
    });

return new Response(readable, { headers: { "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": "GET, POST", "Access-Control-Allow-Headers": "accept, content-type", "Content-Type": "text/event-stream", "Connection": "keep-alive", "Cache-Control": "no-cache", "X-Accel-Buffering": "no", "Content-Encoding": "none", } });

Next.js/Vercel AI/Streaming Issue

Similar Threads

Similar Threads

Similar Threads