since upgrading `@cloudflare/ai` from 1.0.53 to 1.1.0, using the `stream: true` option when invoking

since upgrading

@cloudflare/ai

@cloudflare/ai

from 1.0.53 to 1.1.0, using the

stream: true

stream: true

option when invoking

ai.run(…)

ai.run(…)

no longer streams in the response as would be expected. instead, there are no messages for the most part until the very end when suddenly the entirety of the response streams in at once (still broken into chunks but in extremely rapid succession). here’s how i am using it:

    const ai = new Ai(env.AI);
    const model = '@cf/openchat/openchat-3.5-0106';
    const stream = await ai.run(model, {
        messages: getMessagesFromPrompt({ model, prompt }),
        stream: true,
    });

    return new Response(stream, {
        headers: { 'content-type': 'text/event-stream' },
    });

    const ai = new Ai(env.AI);
    const model = '@cf/openchat/openchat-3.5-0106';
    const stream = await ai.run(model, {
        messages: getMessagesFromPrompt({ model, prompt }),
        stream: true,
    });

    return new Response(stream, {
        headers: { 'content-type': 'text/event-stream' },
    });

    const ai = new Ai(env.AI);
    const model = '@cf/openchat/openchat-3.5-0106';
    const stream = await ai.run(model, {
        messages: getMessagesFromPrompt({ model, prompt }),
        stream: true,
    });

    return new Response(stream, {
        headers: { 'content-type': 'text/event-stream' },
    });

    const ai = new Ai(env.AI);
    const model = '@cf/openchat/openchat-3.5-0106';
    const stream = await ai.run(model, {
        messages: getMessagesFromPrompt({ model, prompt }),
        stream: true,
    });

    return new Response(stream, {
        headers: { 'content-type': 'text/event-stream' },
    });

Marc•3/13/24, 7:25 PM

Is there any timeline for when a multilingual embedding model will be added? It's a requirement for our use case and the only thing stopping us from using this product

Mmichelle do you have a model in mind?

Marc•3/14/24, 2:14 AM

Honestly nothing to specific. We just need to handle text that is potentially in different languages. We are currently using this model:
https://huggingface.co/intfloat/multilingual-e5-small

intfloat/multilingual-e5-small · Hugging Face

Isaac McFadyen•3/14/24, 12:03 PM

I believe BGE-3M was also mentioned here at one point.

Isaac McFadyen•3/14/24, 12:03 PM

https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/BGE_M3

Aacusti since upgrading `@cloudflare/ai` from 1.0.53 to 1.1.0, using the `stream: true` ...

�

😈 Donkey 💫•3/16/24, 5:00 AM

I see 1.1.10 come with massive bugs

alukach•3/17/24, 7:37 AM

Have others been able to run text-to-image models in NextJS apps that run on Pages? I have a nextJS app set up via C3 and a server action performing the following:

"use server";

import { getRequestContext } from "@cloudflare/next-on-pages";
import { Ai } from "@cloudflare/ai";


export async function createThread(prevState: any, formData: FormData) {
  const { env } = getRequestContext();
  const ai = new Ai(env.AI);

  const title = formData.get("title");
  const response = await ai.run(
    "@cf/stabilityai/stable-diffusion-xl-base-1.0",
    {
      prompt: title as string,
    }
  );

"use server";

import { getRequestContext } from "@cloudflare/next-on-pages";
import { Ai } from "@cloudflare/ai";


export async function createThread(prevState: any, formData: FormData) {
  const { env } = getRequestContext();
  const ai = new Ai(env.AI);

  const title = formData.get("title");
  const response = await ai.run(
    "@cf/stabilityai/stable-diffusion-xl-base-1.0",
    {
      prompt: title as string,
    }
  );

However, I'm running into

Error: A Node.js API is used (DecompressionStream) which is not supported in the Edge Runtime. Learn more: https://nextjs.org/docs/api-reference/edge-runtime

Error: A Node.js API is used (DecompressionStream) which is not supported in the Edge Runtime. Learn more: https://nextjs.org/docs/api-reference/edge-runtime

I'm guessing there some decompression going on under the hood with the AI interface?

If it matters, my next step is to write the response as a file to R2, so I'm not sure that I actually need to decompress for my purposes...

Mmichelle any bugs other than this one? we were able to reproduce this one and looking at ...

�

😈 Donkey 💫•3/17/24, 4:12 PM

This is how I generate image:

const response: AiTextToImageOutput = await ai.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input);

const response: AiTextToImageOutput = await ai.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input);

then put in R2

env.MY_BUCKET.put(newImageName, response)

env.MY_BUCKET.put(newImageName, response)

it works in 1.0.53
but It takes typeError in 1.1.0 when putting R2, with something about readable Stream, sorry about unclear error information because I rolled back version 1.0.53

�😈 Donkey 💫This is how I generate image: ```ts const response: AiTextToImageOutput = await...

acustiOP•3/17/24, 5:12 PM

it should be possible to improve the built-in

@cloudflare/ai

@cloudflare/ai

types to narrow the return type based on the received

stream

stream

variable (

stream: true

stream: true

means

ReadableStream<any>

ReadableStream<any>

stream: false

stream: false

means

AiTextToImageOutput

AiTextToImageOutput

or the like), which would remove the need for this workaround.

that said, you can resolve the typescript errors by casting the return value instead of defining the variable type. so in your example, you would do:

const response = (await ai.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input)) as AiTextToImageOutput;

const response = (await ai.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", input)) as AiTextToImageOutput;

if instead you were passing in

stream: true

stream: true

to the

ai.run(…)

ai.run(…)

function, you would want to cast the result to

ReadableStream<any>

ReadableStream<any>

const response = (await ai.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", {...input, stream: true})) as ReadableStream<any>;

const response = (await ai.run("@cf/stabilityai/stable-diffusion-xl-base-1.0", {...input, stream: true})) as ReadableStream<any>;

pups•3/18/24, 4:29 PM

any plans to launch a TTS model as well? like Coqui?

chand1012•3/20/24, 12:52 PM

Hello! I was wonder if some features were on the roadmap, and if not, I'd like to request them! I think that enforcing JSON output like Ollama and Llama.cpp do would be a great feature for devs who want to get structured data output consistently for parsing. Ollama lets you use

format: "json"

format: "json"

in your request and it handles applying the grammar for you. See here for an example: https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-completion

Another features I'd like to see is the option to return token counts along with the response. Its hard to judge what my users are using with no token counts, and I don't really want to have to add a step in order to calculate them.

GitHub

ollama/docs/api.md at main · ollama/ollama

Get up and running with Llama 2, Mistral, Gemma, and other large language models. - ollama/ollama

Raylight•3/21/24, 11:22 AM

Is the

strength

strength

parameter for

@cf/runwayml/stable-diffusion-v1-5-img2img

@cf/runwayml/stable-diffusion-v1-5-img2img

clamped and/or quantized in some weird way? Strength 0.095 generates an output that's close to identical to the source (albeit degraded), while 0.095001 generates a noisy version of the source. Strength 0.0 seems to produce as wildly different output as if it was set to 1.0. (Values outside the range 0.0 to 1.0 throws InferenceUpstreamError.)

Raylight•3/21/24, 11:37 AM

While I'm at it, looks like

@cf/runwayml/stable-diffusion-v1-5-inpainting

@cf/runwayml/stable-diffusion-v1-5-inpainting

throws InferenceUpstreamError on strength values below 0.05 (except 0). Haven't tested this on version 1.1.0, though.

Cchand1012 Hello! I was wonder if some features were on the roadmap, and if not, I'd like t...

acustiOP•3/22/24, 2:53 AM

@chand1012 i had this exact need (the

format: 'json'

format: 'json'

option), and i couldn’t find a satisfactory solution out there, so i made a

parseAsJSON

parseAsJSON

util that takes a string and parses it as JSON even if it has syntax errors or too many closing curly braces, or not enough curly braces, or overly escaped characters, or unescaped characters, or pre- and post-ambles, or…

it’s a JS (typescript) package available on NPM called

@acusti/parsing

@acusti/parsing

. here’s the readme: https://github.com/acusti/uikit/tree/main/packages/parsing

i also implemented some simple few-shot prompting modelled on what’s documented in this guide to get the LLM to return JSON: https://www.pinecone.io/learn/llama-2/#Llama-2-Chat-Prompt-Structure

GitHub

uikit/packages/parsing at main · acusti/uikit

UI toolkit monorepo containing a React component library, UI utilities, a drag-and-drop library, and more - acusti/uikit

Llama 2: AI Developers Handbook | Pinecone

Llama 2 is the latest Large Language Model (LLM) from Meta AI. It has been released as an open-access model, enabling unrestricted access to corporations and open-source hackers alike. Here we learn how to use it with Hugging Face, LangChain, and as a conversational agent.

TheMightyPenguin•3/22/24, 4:05 AM

Hi all, I am trying to use cloudflare AI locally, running

wrangler dev

wrangler dev

, and I get the following error when I try using the

'@cf/facebook/bart-large-cnn'

'@cf/facebook/bart-large-cnn'

model:

Uncaught (async) Error: InferenceUpstreamError: {"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}

Uncaught (async) Error: InferenceUpstreamError: {"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}

If I run

wrangler dev --remote

wrangler dev --remote

, it does work, but I cannot use remote mode as I also use Cloudflare Queues in my app together with workers AI, and queues do not work in remote mode.

Any ideas what might be wrong? I am using

"wrangler": "^3.36.0"

"wrangler": "^3.36.0"

and

"@cloudflare/ai": "^1.1.0"

"@cloudflare/ai": "^1.1.0"

acustiOP•3/22/24, 3:02 PM

@TheMightyPenguin whenever i see

Authentication error

Authentication error

i run

wrangler login

wrangler login

. it surprises me how often i have to re-authenticate when doing local dev (once every few days). i’ve speculated that it might have to do with how often i’m connecting to a different wifi network and changing my IP address, but that might be totally wrong. regardless, re-authing has always resolved it for me.

Aacusti @TheMightyPenguin whenever i see `Authentication error` i run `wrangler login`. ...

TheMightyPenguin•3/22/24, 5:28 PM

that did it, thank you! (I also had to upgrade to the latest version of wrangler, was a few minors behind and wrangler login was showing an error)

CCraig Dennis I'm working on that...unlaunched yet, but let me know if this is along the lines...

fakebrin•3/24/24, 7:40 PM

I seem to get Internal Server Error regardless of the models used. Your (?) public deployment works though

Ffakebrin I seem to get Internal Server Error regardless of the models used. Your (?) publ...

fakebrin•3/24/24, 8:06 PM

I get these logs:

POST https://***.pages.dev/api/chat - Ok @ 3/24/2024, 9:34:08 PM
  (error) Error: Ai binding is undefined. Please provide a valid binding.

POST https://***.pages.dev/api/chat - Ok @ 3/24/2024, 9:34:08 PM
  (error) Error: Ai binding is undefined. Please provide a valid binding.

Ffakebrin I get these logs: ``` POST https://***.pages.dev/api/chat - Ok @ 3/24/2024, 9:34...

Mangum•3/25/24, 2:54 AM

To bind Workers AI to your worker, add the following to the end of your wrangler.toml file:

[ai]
binding = "AI" #available in your worker via env.AI
source: https://blog.cloudflare.com/workers-ai/

The Cloudflare Blog

Workers AI: serverless GPU-powered inference on Cloudflare’s global...

We are excited to launch Workers AI - an AI inference as a service platform, empowering developers to run AI models with just a few lines of code, all powered by our global network of GPUs

MMangum To bind Workers AI to your worker, add the following to the end of your wrangler...

rob•3/25/24, 3:42 AM

yep, the extra step required need to have it bound in your toml file

Mangum•3/25/24, 5:30 AM

Hi there, i'm trying to follow this tutorial https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/
When I enter the command "npx wrangler vectorize create vector-index --dimensions=768 --metric=cosine"
into my terminal, I get: vectorize.not_entitled [code: 1005]
Is this because I'm using a free account and Vectorize is not provisioned?

Cloudflare Docs

Build a Retrieval Augmented Generation (RAG) AI · Cloudflare Worker...

This guide will instruct you through setting up and deploying your first application with Cloudflare AI. You will build a fully-featured AI-powered …

MMangum To bind Workers AI to your worker, add the following to the end of your wrangler...

fakebrin•3/25/24, 7:07 AM

it is there in the wrangler.toml file, if the clone worked correctly

zehawk•3/25/24, 8:51 AM

Unable to run even the simplest code on wrangler dev local (eg https://developers.cloudflare.com/workers-ai/configuration/workers-ai-sdk/). Getting

[wrangler:err] InferenceUpstreamError: {"errors":[{"message":"Server Error","code":1000}],"success":false,"result":{},"messages":[]}

[wrangler:err] InferenceUpstreamError: {"errors":[{"message":"Server Error","code":1000}],"success":false,"result":{},"messages":[]}

AI workers were working fine earlier, and starting giving an error now. Hence testing with boiler plate code.

Raylight•3/25/24, 5:49 PM

Some feedback regarding

@cf/runwayml/stable-diffusion-v1-5-inpainting

@cf/runwayml/stable-diffusion-v1-5-inpainting

(and

@cf/runwayml/stable-diffusion-v1-5-img2img

@cf/runwayml/stable-diffusion-v1-5-img2img

):
The models seem to produce an output with a noticeable shift in color balance. Repeated inpainting will quickly shift the image towards purple. See the attached images below. The first image is the result of doing 10 passes (i.e. using the result as source) of inpainting with an empty (black) mask. The second is the same process but with an added "hack" (i.e. more of a band-aid than a solution) that does histogram matching after each pass. The third image is a typical result from a single pass of inpainting merged with unmasked pixels from the source. (Shown with mask overlayed on source.)

This seems to be a well-known problem that (possibly) can be traced back to the VAE, and (possibly) be solved by using an updated version of the VAE. Whether that's the case here is another question.

inpainting_stp20_g7.5_str1_10_iter_w_histo_matching.png

Raylight•3/25/24, 5:49 PM

(The black to the left is simply because the "safety checker" believes the image is harmful content without it.)

Mmichelle this is so interesting, is it a model problem though or specific to workers ai?

Raylight•3/25/24, 10:38 PM

I believe it's a problem with the model, so maybe it belongs to the catch all "cloudflare-ai". The result should be the same if using the API. Although using workers ai + client side javascript will involve more head-scratching and second-guessing your code.

I found this https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/410 which seems to match my own testing.

playlogo•3/25/24, 10:43 PM

Hey, is there any way we can use

mistral-7b-instruct-v0.2

mistral-7b-instruct-v0.2

via the workers rest api ?

playlogo•3/25/24, 10:44 PM

Also I'm sometimes getting the same output for slightly different promps. Is this some sort of caching that i can maybe disable ?

rob•3/25/24, 11:00 PM

are you using the ai gateway ?

Dan (Openside)•3/26/24, 1:47 AM

Im using model

@hf/thebloke/deepseek-coder-6.7b-instruct-awq

@hf/thebloke/deepseek-coder-6.7b-instruct-awq

It appears there is a character limit on the response (around 800 chars). Is this a known limit? any way to get the full response?

playlogo•3/26/24, 11:57 AM

According to their API Docs you can set the

max_tokens

max_tokens

property in the request. But don't see it mentioned anywhere else, so it might not work...

Cloudflare API Documentation

Interact with Cloudflare's products and services via the Cloudflare API

YeungKC•3/26/24, 12:45 PM

Hey guys, I would like to integrate an AI interface into worker-rs. Does anyone know where I can find the source code for @cloudflare/ai?

rob•3/26/24, 1:33 PM

https://twitter.com/StabilityAI/status/1772345514023116828

Stability AI (@StabilityAI) on X

Introducing Stable Code Instruct 3B, our new instruction tuned LLM based on Stable Code 3B. With natural language prompting, this model can handle a variety of tasks such as code generation, math and other software engineering related outputs.

This model’s performance rivals…

Twitter

•

3/25/24, 7:31 PM

YYeungKC Hey guys, I would like to integrate an AI interface into worker-rs. Does anyone ...

John Spurlock•3/26/24, 9:51 PM

it's not public afaik, you can always: https://www.npmjs.com/package/@cloudflare/ai?activeTab=code

npm

@cloudflare/ai

Cloudflare's Workers AI SDK. Latest version: 1.1.0, last published: 19 days ago. Start using @cloudflare/ai in your project by running

npm i @cloudflare/ai

npm i @cloudflare/ai

. There are 3 other projects in the npm registry using @cloudflare/ai.

John Spurlock•3/26/24, 9:52 PM

or install locally etc

JJohn Spurlock it's not public afaik, you can always: https://www.npmjs.com/package/@cloudflare...

YeungKC•3/27/24, 1:45 AM

Is it not ready for open source yet, or is there no plan to open source it?

James•3/27/24, 1:53 AM

I believe the goal is to make AI a more native binding in workers and remove the need for the package entirely

YeungKC•3/27/24, 1:58 AM

Thx, I'm really looking forward to it.

rob•3/27/24, 1:58 AM

yah more for calling inside of workers

Nooc•3/28/24, 3:55 AM

I'm calling Workers AI in a nextjs project, the project used the edge runtime in every page, because I want to deploy it on Cloudflare Pages. But the Workers AI calling continuously occur the error '⨯ Error: A Node.js API is used (DecompressionStream) which is not supported in the Edge Runtime.'. Has anyone had the same problem?

Nooc•3/28/24, 3:55 AM

since upgrading `@cloudflare/ai` from 1.0.53 to 1.1.0, using the `stream: true` option when invoking

Similar Threads

Similar Threads

Similar Threads