Wrap the loop in an async function then put it in a waitUntil

B

bkyerv•9/19/23, 12:43 PM

@m.a.t.t.t it does, but the way it's set up essentially negates the benefits of streaming. The issue is that it waits for all chunks from the OpenAI completion to be written into the stream before sending the entire response to the client. This behavior is contrary to the expectations of users, including myself and another individual who opened a ticket in the help section. We anticipate that the response should be streamed to the client in smaller, incremental pieces. To achieve this, the following changes need to be made:

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

HHello, I’m Allie!Wrap the loop in an async function then put it in a waitUntil

B

bkyerv•9/19/23, 12:45 PM

Thank you, that's what I ended up doing by talking to chatgpt)) In general, can you please suggest some resources to learn more about streams. Not in million years I would guess that response needs to be constructed before starting to write into stream. Also what is the need for waitUntil?

Bbkyerv Thank you, that's what I ended up doing by talking to chatgpt)) In general, can ...

H

Hello, I’m Allie!OP•9/19/23, 12:45 PM

If you don’t use waitUntil then it will cut off the loop once you return a Response

S

sathoro•9/19/23, 1:02 PM

yeah that doc (https://developers.cloudflare.com/workers/examples/openai-sdk-streaming/) needs to be updated lol

HHello, I’m Allie!Wrap the loop in an async function then put it in a waitUntil

S

sathoro•9/19/23, 1:03 PM

this is what we do in prod, works great

E

Elliot Hesp•9/19/23, 2:01 PM

When using the cache within a worker, does it autoamtically apply a cache-control header if cached? I don't set one when I store it in cache, but there seems to be one there with a value of

✦

✦ xcaeser•9/19/23, 2:50 PM

tried workers for the first time. usually on aws lambda and vercel functions...

✦

✦ xcaeser•9/19/23, 2:51 PM

this is vercel

✦

✦ xcaeser•9/19/23, 2:51 PM

this is aws

✦

✦ xcaeser•9/19/23, 2:51 PM

and this workers LMAO

✦✦ xcaeser and this workers LMAO

H

Hello, I’m Allie!OP•9/19/23, 2:52 PM

What does your code do?

✦

✦ xcaeser•9/19/23, 2:52 PM

same code, nodejs, TypeScript on vercel and workers, mjs on aws (1GB disk + 1GB RAM)

HHello, I’m Allie!What does your code do?

✦

✦ xcaeser•9/19/23, 2:52 PM

uses cloudflare KV,

✦

✦ xcaeser•9/19/23, 2:52 PM

just basic CRUD

✦

✦ xcaeser•9/19/23, 2:53 PM

vercel and aws use vercel kv (which is upstash)

✦✦ xcaeser uses cloudflare KV,

H

Hello, I’m Allie!OP•9/19/23, 2:53 PM

What does it show with a hot read?

H

Hello, I’m Allie!OP•9/19/23, 2:54 PM

Also, wouldn't this more be a comparison of CF KV vs. Upstash?

H

Hello, I’m Allie!OP•9/19/23, 2:54 PM

Since the code is just doing reads no matter what?

H

Hello, I’m Allie!OP•9/19/23, 2:54 PM

Wonder if it were faster/slower if you read from Upstash with Workers...

✦

✦ xcaeser•9/19/23, 2:55 PM

what's behind cloudflare kv?

✦

✦ xcaeser•9/19/23, 2:55 PM

i'll just put these here

✦

✦ xcaeser•9/19/23, 2:56 PM

again, first time trying out cloudflare, but hard to be aws

✦✦ xcaeser what's behind cloudflare kv?

H

Hello, I’m Allie!OP•9/19/23, 2:56 PM

Proprietary storage engine afaik

HHello, I’m Allie!Wonder if it were faster/slower if you read from Upstash with Workers...

✦

✦ xcaeser•9/19/23, 2:57 PM

shouldn't KV be optimized for workers? and maybe closer in region?

HHello, I’m Allie!Proprietary storage engine afaik

✦

✦ xcaeser•9/19/23, 2:57 PM

should've gone for foundation db

✦✦ xcaeser shouldn't KV be optimized for workers? and maybe closer in region?

H

Hello, I’m Allie!OP•9/19/23, 2:57 PM

There is some work being done on it, but right now it is dual-zone Europe and Americas with a cache

H

Hello, I’m Allie!OP•9/19/23, 2:57 PM

So if you aren’t in the US/EU, you might get slower reads

✦

✦ xcaeser•9/19/23, 2:58 PM

I'm in paris

✦

✦ xcaeser•9/19/23, 2:59 PM

developer exp is top notch tho (from terminal as well), way better than raw aws

✦✦ xcaeser should've gone for foundation db

H

Hello, I’m Allie!OP•9/19/23, 3:00 PM

FoundationDB(Apache licensed) was released just 5 months before KV, so it might have been that they started work on it before it was viable

H

Hello, I’m Allie!OP•9/19/23, 3:01 PM

Though I would note too that Vercel Edge is Workers too

H

Hello, I’m Allie!OP•9/19/23, 3:01 PM

So for any non-platform-bound applications, they should perform about the same

HHello, I’m Allie!Though I would note too that Vercel Edge is Workers too

✦

✦ xcaeser•9/19/23, 3:01 PM

nop it's aws afaik

✦✦ xcaeser nop it's aws afaik

H

Hello, I’m Allie!OP•9/19/23, 3:05 PM

That’s their core(I.E. non-Edge). That’s also why they are so particular about the APIs that the Edge Runtime does and doesn’t support

H

Hello, I’m Allie!OP•9/19/23, 3:05 PM

Because the Edge Runtime(WorkerD) is V8, but not Node

H

Hello, I’m Allie!OP•9/19/23, 3:06 PM

https://news.ycombinator.com/item?id=29003514

leerob

Vercel's Edge Functions are built on top of Cloudflare Workers. The runtime is designed for any similar provider, but we've chosen Cloudflare for now because they have an amazing product.

Hacker News

•

10/26/21, 5:41 PM

HHello, I’m Allie!https://news.ycombinator.com/item?id=29003514

✦

✦ xcaeser•9/19/23, 3:09 PM

thx for sharing, reading now

✦

✦ xcaeser•9/19/23, 3:10 PM

okay let me try the worker with vercel kv in this case and see if it's better

✦

✦ xcaeser•9/19/23, 3:29 PM

how to get around this? Uncaught ReferenceError: process is not defined
cloudflare workers don't use node right?

✦

✦ xcaeser•9/19/23, 3:34 PM

nvm, followed upstash cloudfare quickstart

HHello, I’m Allie!Also, wouldn't this more be a comparison of CF KV vs. Upstash?

✦

✦ xcaeser•9/19/23, 3:37 PM

you're right, switched to upstach. Cloudflare KV is very very slow

✦

✦ xcaeser•9/19/23, 3:41 PM

still aw seems to be faster for now;

U

Unsmart•9/19/23, 3:50 PM

you show 2 responses for the other platform one of which is faster one of which is slower so

✦✦ xcaeser still aw seems to be faster for now;

C

Chaika•9/19/23, 3:52 PM

What are you measuring? Puts or Gets?

should've gone for foundation db

The magic with KV is its unlimited scale and getting faster the more requests you give it, it's made for read-heavy cases, like for example Cloudflare Pages uses it
https://developers.cloudflare.com/kv/learning/how-kv-works/

You can signal for it to cache results longer as well, and stream back responses, ex

   let value = await env.KV.get("index.html", {cacheTtl: 86400, type: "stream" });
    return new Response(value, {
      headers: {
        "content-type": "text/html"
      }
    })

   let value = await env.KV.get("index.html", {cacheTtl: 86400, type: "stream" });
    return new Response(value, {
      headers: {
        "content-type": "text/html"
      }
    })

   let value = await env.KV.get("index.html", {cacheTtl: 86400, type: "stream" });
    return new Response(value, {
      headers: {
        "content-type": "text/html"
      }
    })

   let value = await env.KV.get("index.html", {cacheTtl: 86400, type: "stream" });
    return new Response(value, {
      headers: {
        "content-type": "text/html"
      }
    })

(It is eventually consistent by design, but by getting it to cache for longer you're not going to see updates until cache expires or if you write in same location. (KV 2.0 which was delayed aimed to solve this by checking every minute in the background for updates, but it was delayed due to issues))
It's pretty fast, once its cached locally ~20ms or less

CChaika What are you measuring? Puts or Gets? > should've gone for foundation db The mag...

✦

✦ xcaeser•9/19/23, 5:00 PM

understood thx chaika

D

dave•9/20/23, 5:16 AM

What are your preferred ways of securely accessing a Worker route from an AWS Lambda? I’ve done the reverse before, undecided on what makes the most sense for the other way around.

Ddave What are your preferred ways of securely accessing a Worker route from an AWS La...

S

sathoro•9/20/23, 7:33 AM

good old secret key in the headers

Ddave What are your preferred ways of securely accessing a Worker route from an AWS La...

H

Hello, I’m Allie!OP•9/20/23, 7:36 AM

mTLS?

A

Amnon•9/20/23, 8:07 AM

Hey,
I'm using Workers to push some logic to the request and proxying the request to the origin backend
When combining that with the Dynamic URL Rewrite Transform Rule it seems like the Transform Rule is triggered two times - from the client to the worker and when proxying the request
In my Dynamic Transform Rule, I used -

concat(“/v1”, http.request.uri.path)

concat(“/v1”, http.request.uri.path)

concat(“/v1”, http.request.uri.path)

concat(“/v1”, http.request.uri.path) and ended up with

/v1/v1/<original_req_path>

/v1/v1/<original_req_path>

/v1/v1/<original_req_path>

/v1/v1/<original_req_path>

I tried to adjust the Custom Filter Expression with no success (skipping the rule every time the URL starts with

/v1

/v1

/v1

/v1 works but doesn't feel robust enough)
Any idea how to overcome this behavior?
Thanks

(also posted to the community - https://community.cloudflare.com/t/proxyrequest-in-workers-triggers-transform-rules-more-than-once/557777)

Wrap the loop in an async function then put it in a waitUntil

Similar Threads

Similar Threads

Similar Threads