Thread on Workers AI onboarding for a

James · 2024-04-06T16:46:13.754Z

Thread on Workers AI onboarding for a new user 🧵

J

JamesOP•4/6/24, 4:46 PM

I just put together this worker after following the docs:

import { Ai } from '@cloudflare/ai'

interface Env {
    AI: Fetcher;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx }});

        const number = Math.random();
        const response = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: `I have ${number} apples. How many apples do I have?"`,
            stream: true
        });

        return new Response(response);
    },
};

import { Ai } from '@cloudflare/ai'

interface Env {
    AI: Fetcher;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx }});

        const number = Math.random();
        const response = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: `I have ${number} apples. How many apples do I have?"`,
            stream: true
        });

        return new Response(response);
    },
};

import { Ai } from '@cloudflare/ai'

interface Env {
    AI: Fetcher;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx }});

        const number = Math.random();
        const response = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: `I have ${number} apples. How many apples do I have?"`,
            stream: true
        });

        return new Response(response);
    },
};

import { Ai } from '@cloudflare/ai'

interface Env {
    AI: Fetcher;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext) {
        const ai = new Ai(env.AI, { sessionOptions: { ctx }});

        const number = Math.random();
        const response = await ai.run('@cf/mistral/mistral-7b-instruct-v0.1', {
            prompt: `I have ${number} apples. How many apples do I have?"`,
            stream: true
        });

        return new Response(response);
    },
};

From the docs here: https://developers.cloudflare.com/workers-ai/configuration/workers-ai-sdk/#async-airun

J

JamesOP•4/6/24, 4:47 PM

Immediately, I am hit with two type errors:

-

Argument of type 'AiTextGenerationOutput | ReadableStream<any>' is not assignable to parameter of type 'BodyInit | null | undefined'.  Type '{ response?: string | undefined; }' is not assignable to type 'BodyInit | null | undefined'.

Argument of type 'AiTextGenerationOutput | ReadableStream<any>' is not assignable to parameter of type 'BodyInit | null | undefined'.  Type '{ response?: string | undefined; }' is not assignable to type 'BodyInit | null | undefined'.

Argument of type 'AiTextGenerationOutput | ReadableStream<any>' is not assignable to parameter of type 'BodyInit | null | undefined'.  Type '{ response?: string | undefined; }' is not assignable to type 'BodyInit | null | undefined'.

Argument of type 'AiTextGenerationOutput | ReadableStream<any>' is not assignable to parameter of type 'BodyInit | null | undefined'.  Type '{ response?: string | undefined; }' is not assignable to type 'BodyInit | null | undefined'.

-

Object literal may only specify known properties, and 'sessionOptions' does not exist in type 'AiOptions'.

Object literal may only specify known properties, and 'sessionOptions' does not exist in type 'AiOptions'.

Object literal may only specify known properties, and 'sessionOptions' does not exist in type 'AiOptions'.

Object literal may only specify known properties, and 'sessionOptions' does not exist in type 'AiOptions'.

J

JamesOP•4/6/24, 4:47 PM

sessionOptions

sessionOptions

sessionOptions

sessionOptions appears to not be a thing anymore, since it looks like

1.1.0

1.1.0

1.1.0

1.1.0, released almost a month ago. And the other error seems to just be bad types/code in the docs.

J

JamesOP•4/6/24, 4:48 PM

My code "works" (https://hello-ai.jross.workers.dev/), but my DX here is super rough.

J

JamesOP•4/6/24, 4:48 PM

I then also saw a few random 503s when developing, which went away after a refresh.

J

JamesOP•4/6/24, 4:49 PM

Frankly, at this point, as a new user, I'm kinda done.

Type errors, random response failures, docs not matching reality. This does not feel like it deserves a "GA" title right now.

J

JamesOP•4/6/24, 4:49 PM

cc @celso @michelle @Logan Grasby @Craig

V

Victor•4/6/24, 4:51 PM

I also want to chime in that the types for image-gen specify a

UInt8Array

UInt8Array

UInt8Array

UInt8Array being returned, but in some cases, it seems to be the raw

ReadableStream<UInt8Array>

ReadableStream<UInt8Array>

ReadableStream<UInt8Array>

ReadableStream<UInt8Array> from the underlying

Fetcher

Fetcher

Fetcher

Fetcher that is used.

The docs show it being returned immediately inside a

Response

Response

Response

Response object which can take either without any issue or error. But it is important if you want to do any other work that directly re-stream it back out the worker.

VVictor I also want to chime in that the types for image-gen specify a `UInt8Array` bein...

V

Victor•4/6/24, 4:51 PM

My 2 cents would be to carry over the

stream

stream

option from text-gen for instances where you return directly a

ReadableStream

ReadableStream

ReadableStream

ReadableStream to remove the buffering (if desired)?

C

Craig Dennis•4/8/24, 5:22 PM

Thank you for the feedback @James , looking into these!

C

Craig Dennis•4/8/24, 5:24 PM

And you as well @Victor @DemosJarco | Chainfuse

Thread on Workers AI onboarding for a

Similar Threads

Similar Threads

Similar Threads