ok thanks! for context, im trying to build something similar to durable objects I tried DO's but per

U

Unsmart•9/19/23, 3:30 AM

I’m not sure they even really offer that to enterprise customers beyond maybe jurisdictions but also using global scope is something suggested not to do

M

mattOP•9/19/23, 3:32 AM

It seems to be recommended here:
https://developers.cloudflare.com/workers/platform/limits/#memory

Use global variables to persist data between requests on individual nodes

M

mattOP•9/19/23, 3:33 AM

There is also a thread about the ability to route to a specific worker DC colo: https://community.cloudflare.com/t/how-to-run-workers-on-specific-datacenter-colos/385851

M

mattOP•9/19/23, 3:35 AM

No wait, it's this one:
https://community.cloudflare.com/t/explicitly-choosing-worker-location-to-perform-request-based-on-path/383886

�

💻🐒🔫🔥🤖•9/19/23, 3:39 AM

Is it possible to make auto-deployments only trigger when a pull request is made on a github branch? As opposed to on every single commit?

I

Izrath•9/19/23, 3:40 AM

Im building out a discord bot which requires a response in 3 seconds. My function will take longer than that so I need to execute a separate worker separately to run my function and update discord when its done. Is it do-able to call another worker and then return/end my current worker?

K

kian•9/19/23, 3:42 AM

You return a deferred response and use waitUntil to patch that message

K

kian•9/19/23, 3:43 AM

https://developers.cloudflare.com/workers/runtime-apis/fetch-event/#waituntil

FetchEvent · Cloudflare Workers docs

In Workers, any incoming HTTP requests are referred to as "fetch" events. A Worker will respond to the HTTP request with the handler method that was …

I

Izrath•9/19/23, 3:56 AM

Thanks, the example I am building off of uses itty-router, which I think uses the ES modules format. This article helps, I think I was just missing the context variable. Thanks!

Mmatt ok thanks! for context, im trying to build something similar to durable objects ...

H

Hello, I’m Allie!•9/19/23, 7:24 AM

I would note too that even requests from the same device(and not the same app) can be routed to two different datacenters, so you aren’t guaranteed to ever hit the same one twice

HHello, I’m Allie!I would note too that even requests from the same device(and not the same app) c...

M

mattOP•9/19/23, 10:24 AM

In aware - see the community thread I linked to where it’s mentioned that you can route requests to a specific Colo after it hits the first worker

Mmatt In aware - see the community thread I linked to where it’s mentioned that you ca...

H

Hello, I’m Allie!•9/19/23, 10:28 AM

As far as I can tell with that specific demo, it will route you to a specific Colo(which may contain multiple datacenters). And even then, you still aren't guaranteed to hit the same physical machine(or isolate, depending on traffic levels), which would be required to be able to reuse the global scope.

B

bkyerv•9/19/23, 10:33 AM

hello everyone, I am working on streaming text data from an OpenAI endpoint to my client while maintaining the stream's continuous nature. However, I keep encountering an error that suggests the ReadableStream is expected to handle byte-like data. Is there a way to configure TransformStream to work with text or string data instead?

B

bkyerv•9/19/23, 10:34 AM

Uncaught (in promise) TypeError: This ReadableStream did not return bytes.

Uncaught (in promise) TypeError: This ReadableStream did not return bytes.

Bbkyerv hello everyone, I am working on streaming text data from an OpenAI endpoint to m...

H

Hello, I’m Allie!•9/19/23, 10:34 AM

Can't you convert to-from bytes with

Text(En/De)coder

Text(En/De)coder

?

B

bkyerv•9/19/23, 10:35 AM

I thought if openai returns chunks in string I can do the same. Are you suggesting to convert the string chunks I received from openai to bytes before transmitting to client?

Bbkyerv I thought if openai returns chunks in string I can do the same. Are you suggesti...

H

Hello, I’m Allie!•9/19/23, 10:36 AM

Maybe? Try it, and see if it works

B

bkyerv•9/19/23, 10:43 AM

it works but when on the client I am logging the chunks from the reader I see the whole text logged in one go instead of piece by piece. does transformstream implementation in cloudflare performs any buffering before sending the data over network?

B

bkyerv•9/19/23, 10:43 AM

thank you for pointing the text encoder

B

bkyerv•9/19/23, 11:39 AM

I just noticed someone else got stuck into it as well 2 days ago workers-and-pages-helpHow to make a Worker stream (an OpenAI) response back to the front-end?

Bbkyerv I just noticed someone else got stuck into it as well 2 days ago https://discord...

M

mattOP•9/19/23, 11:46 AM

Are you sure you can call

writer.close()

writer.close()

before the response has been received?

B

bkyerv•9/19/23, 12:05 PM

yes, but I call it outside of the loop that iterates over chunks like so

const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();

                    try {
                        for await (const chunk of completions) {
                            const content = chunk['choices'][0].delta.content!;
                            const encodedMessage = encoder.encode(content);
                            writer.write(encodedMessage);
                        }
                        writer.close();
                        return new Response(readable, {
                            headers: {
                                ...corsHeaders,
                            },
                        });
                    } catch (e) {
                        console.error(e);
                        // writer.abort(e);
                        return new Response('bad bad error', { headers: corsHeaders });
                    }

const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();

                    try {
                        for await (const chunk of completions) {
                            const content = chunk['choices'][0].delta.content!;
                            const encodedMessage = encoder.encode(content);
                            writer.write(encodedMessage);
                        }
                        writer.close();
                        return new Response(readable, {
                            headers: {
                                ...corsHeaders,
                            },
                        });
                    } catch (e) {
                        console.error(e);
                        // writer.abort(e);
                        return new Response('bad bad error', { headers: corsHeaders });
                    }

const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();

                    try {
                        for await (const chunk of completions) {
                            const content = chunk['choices'][0].delta.content!;
                            const encodedMessage = encoder.encode(content);
                            writer.write(encodedMessage);
                        }
                        writer.close();
                        return new Response(readable, {
                            headers: {
                                ...corsHeaders,
                            },
                        });
                    } catch (e) {
                        console.error(e);
                        // writer.abort(e);
                        return new Response('bad bad error', { headers: corsHeaders });
                    }

const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();

                    try {
                        for await (const chunk of completions) {
                            const content = chunk['choices'][0].delta.content!;
                            const encodedMessage = encoder.encode(content);
                            writer.write(encodedMessage);
                        }
                        writer.close();
                        return new Response(readable, {
                            headers: {
                                ...corsHeaders,
                            },
                        });
                    } catch (e) {
                        console.error(e);
                        // writer.abort(e);
                        return new Response('bad bad error', { headers: corsHeaders });
                    }

Bbkyerv yes, but I call it outside of the loop that iterates over chunks like so ```Type...

H

Hello, I’m Allie!•9/19/23, 12:06 PM

Oh yeah, you have to return the response before you write to the

WritableStream

WritableStream

H

Hello, I’m Allie!•9/19/23, 12:07 PM

Otherwise it will wait for it to finish before returning

B

bkyerv•9/19/23, 12:09 PM

if I move the return statements above the loop the whole loop will not be executed. Can you please point more precisely where exactly I need tto move the returning the response?

M

mattOP•9/19/23, 12:17 PM

@bkyerv is this not working? https://developers.cloudflare.com/workers/examples/openai-sdk-streaming/

Bbkyerv if I move the return statements above the loop the whole loop will not be execut...

H

Hello, I’m Allie!•9/19/23, 12:42 PM

Wrap the loop in an async function then put it in a waitUntil

B

bkyerv•9/19/23, 12:43 PM

@m.a.t.t.t it does, but the way it's set up essentially negates the benefits of streaming. The issue is that it waits for all chunks from the OpenAI completion to be written into the stream before sending the entire response to the client. This behavior is contrary to the expectations of users, including myself and another individual who opened a ticket in the help section. We anticipate that the response should be streamed to the client in smaller, incremental pieces. To achieve this, the following changes need to be made:

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

Typescript 
const { readable, writable } = new TransformStream();

                    const writer = writable.getWriter();
                    const encoder = new TextEncoder();
                    const response = new Response(readable, {
                        headers: {
                            ...corsHeaders,
                        },
                    });
                    (async () => {
                        try {
                            for await (const chunk of completions) {
                                const content = chunk['choices'][0].delta.content!;
                                const encodedMessage = encoder.encode(content);
                                writer.write(encodedMessage);
                            }
                            writer.close();
                        } catch (e) {
                            console.error(e);
                            // writer.abort(e);
                            return new Response('bad bad error', { headers: corsHeaders });
                        }
                    })();
                    return response;

HHello, I’m Allie!Wrap the loop in an async function then put it in a waitUntil

B

bkyerv•9/19/23, 12:45 PM

Thank you, that's what I ended up doing by talking to chatgpt)) In general, can you please suggest some resources to learn more about streams. Not in million years I would guess that response needs to be constructed before starting to write into stream. Also what is the need for waitUntil?

Bbkyerv Thank you, that's what I ended up doing by talking to chatgpt)) In general, can ...

H

Hello, I’m Allie!•9/19/23, 12:45 PM

If you don’t use waitUntil then it will cut off the loop once you return a Response

S

sathoro•9/19/23, 1:02 PM

yeah that doc (https://developers.cloudflare.com/workers/examples/openai-sdk-streaming/) needs to be updated lol

HHello, I’m Allie!Wrap the loop in an async function then put it in a waitUntil

S

sathoro•9/19/23, 1:03 PM

this is what we do in prod, works great

E

Elliot Hesp•9/19/23, 2:01 PM

When using the cache within a worker, does it autoamtically apply a cache-control header if cached? I don't set one when I store it in cache, but there seems to be one there with a value of

✦

✦ xcaeser•9/19/23, 2:50 PM

tried workers for the first time. usually on aws lambda and vercel functions...

✦

✦ xcaeser•9/19/23, 2:51 PM

this is vercel

✦

✦ xcaeser•9/19/23, 2:51 PM

this is aws