We are considering migrating our existing Azure Whisper integration to use Cloudflare Whisper model

Wwhakamolenz We are considering migrating our existing Azure Whisper integration to use Cloud...

I

Cloudflare's Whisper model is the

base

base

version of the model. Depending on what you're using on Azure it maybe be more accurate (I believe above base there is

small

small

,

medium

medium

, and

large

large

, all of which will be more accurate)

IIsaac McFadyen Cloudflare's Whisper model is the `base` version of the model. Depending on what...

W

whakamolenzOP•5/28/24, 3:01 AM

ok thanks for the info

Mmichelle we are working to make it configurable per request

J

James•5/28/24, 4:16 PM

will it default back to png as documented?

T

Tempo•5/28/24, 8:05 PM

Hi, I have a question about links between AI and R2. I want to use a model for captioning images from R2, do I have to make a fetch in my worker code of the R2 image path before sending it to cf AI or is there a better way?
(example : https://developers.cloudflare.com/workers-ai/models/uform-gen2-qwen-500m/)

K

kNite•5/29/24, 12:19 AM

Hi, I have a problem with Cloudflare Workers + Hono + Cloudflare AI, I have a route defined (i.e. /v1/chat/completions) and when the user triggers that route I try to run env.AI.run([model], [etc]), but I get this error:

TypeError: Cannot read properties of undefined (reading 'AI')

TypeError: Cannot read properties of undefined (reading 'AI')

. This is my Env interface:

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

Also tried removing the

const env = [...]

const env = [...]

and running AI inside a function with

function exampleFunction(env: Env)

function exampleFunction(env: Env)

but it also didn't work.

L

liuxiaohua72•5/29/24, 12:39 AM

Is it possible for workers-ai to support OpenAI format API?

JJames will it default back to png as documented?

J

James•5/30/24, 3:59 PM

any updates here @michelle ? Is this an expected breaking change, or will this be reverted back to png?

N

Nicolas92•5/30/24, 8:14 PM

Hi guys, how can use this worker ai ? Have someone the Java script for it ?

NNicolas92 Hi guys, how can use this worker ai ? Have someone the Java script for it ?

I

Isaac McFadyen•5/30/24, 8:58 PM

You can follow any of these guides. Use CLI or Dashboard if you want to deploy a Worker, use the API one for if you want to access it externally without a Worker: https://developers.cloudflare.com/workers-ai/get-started/

J

js•5/31/24, 5:34 AM

Will llama 3 loras be supported soon? https://developers.cloudflare.com/workers-ai/fine-tunes/loras/

Cloudflare Docs

Using LoRA adapters · Cloudflare Workers AI docs

Upload and use LoRA adapters to get fine-tuned inference on Workers AI.

P

Puliczek•6/1/24, 2:07 PM

Hi, are there any startups that are using "Workers for Platforms"?

PPuliczek Hi, are there any startups that are using "Workers for Platforms"?

I

Isaac McFadyen•6/1/24, 5:04 PM

This would probably be a better question for #workers-for-platforms. This channel is for Workers AI.

Mmichelle sorry - was at Connect! We are working on making it configurable per request, TB...

R

Raylight•6/1/24, 6:58 PM

Hmm.. quite a lot going on with that model right now. Was going to give feedback on the compression level as the immediate issue, but now the model is broken for deployed workers that are using SDK 1.0.53 or earlier. Which is kinda.. more immediate. Feels like the model is actively being worked on right in the open, so I'm thinking.. should I just hold off a bit, or try to play catch up with feedback?

Mmichelle are you using native binding? or the npm install?

R

Raylight•6/1/24, 7:05 PM

I'm using all of them. 1.0.53, 1.1.0, native binding, api.

T

Telion•6/1/24, 7:28 PM

Hi, I was wondering whether there is any way to improve the results of the LLAMA 3 here. I've been trying

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct and

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct and both provide me with almost completely random responses when trying to use a system message along the prompt.

I am trying to classify user prompts by a list of intents. An example of prompt I am currently struggling with is "What would I need to do to learn js?". While example of intents is:
'1 Track progress and view improvements',
'2 Generate a learning plan or prepare learning materials list',
'3 Need an explanation',
'4 Seek motivation',

Here's how I prompt the model with the data:

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

The responses to this are pretty random. I get:
1
4
3
2
3

When I moved the message from the system prompt into user prompt it started working better but the results are still really unreliable. I wanted to check with experienced people here whether my prompting is too unprecise or there is something wrong with how I understand the CF's AI.

The real issue here is that when I just use the same model on HF or Meta or even drop it into gemini or gpt-4o/3.5t all of them return perfect results, straight "2"s but not on CF

T

Telion•6/1/24, 7:30 PM

I've used the same prompt on all of those, the results differ majorly with what I experience on CF's

T

Telion•6/1/24, 7:32 PM

Although when I put the same message into user's prompt and get rid of the system prompt it starts working a lot better. Yet other prompts still fail randomly, while seem to be percieved correctly on other platforms.

system-prompt-seems-to-not-enforce-behavior-v0-z6f8jse1fz3d1.webp

system-prompt-seems-to-not-enforce-behavior-v0-g6geobw7fz3d1.webp

TTelion Hi, I was wondering whether there is any way to improve the results of the LLAMA...

U

Unsmart•6/2/24, 2:41 AM

As explained in the Workers AI docs for each model you want to use scoped prompts with hints (https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct/#scoped-prompts)

Basically instead of a single system message with the task and response type, you have the system message with the AIs goal followed by examples of how you want it to respond by using the user/assistant roles.

The example that CF specifically has in the docs:

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

UUnsmart As explained in the Workers AI docs for each model you want to use scoped prompt...

T

Telion•6/2/24, 3:40 AM

Right, my issue was that scoped prompts don't improve the results, they actually make it worse. I specifically provided an example of scope prompts first, then with the images I demonstrated that without scoped prompts I get better results but they are still not any close to being what LLAMA on HF or Meta.ai provide.

T

Telion•6/2/24, 3:40 AM

The first example with scoped prompts I added above results in absolutely random results while putting everything into user prompts actually improves it by a lot

U

Unsmart•6/2/24, 3:42 AM

The example you showed technically has scoped prompts yes, but not used in a the best way. If you follow the prompting method recommended of

system: you do this
user: an example thing a user might say
assistant: what you want the response to be
user: example 2
assistant: answer 2
user: <actual input>

It will work wonderfully, I have had 0 issues with responses when prompting in that format.

T

Telion•6/2/24, 3:43 AM

Oh, I missed the assistant part. I noticed that not all models support this feature so I skipped it

T

Telion•6/2/24, 3:43 AM

In the example though assistant seems to be dynamic, does it get generated or how do I assign the right assistant prompt?

U

Unsmart•6/2/24, 3:44 AM

Its worked perfectly fine on every model I have used it on, no clue what doesnt support it... maybe the non text based ones

U

Unsmart•6/2/24, 3:45 AM

The assistant prompts are static, you just give it some generic examples. For a case where you want to have intent scoring you probably want to have like 2-3 examples for each category

T

Telion•6/2/24, 3:45 AM

Wait, is the assistant prompt just the history of answers?

T

Telion•6/2/24, 3:45 AM

that the llm made?

U

Unsmart•6/2/24, 3:45 AM

No that you make

U

Unsmart•6/2/24, 3:46 AM

You use the assistant to teach it how to respond

T

Telion•6/2/24, 3:46 AM

oh, wait I see

T

Telion•6/2/24, 3:46 AM

your example actually includes user prompts as part of the general mechanism describing it how ot behave

T

Telion•6/2/24, 3:46 AM

Let me try that, thank you!

T

Telion•6/2/24, 3:47 AM

I didn't think I can or should use user prompts to explain LLM something, I was 100% sure user prompts are jsut the user input and I shouldn't mess with that

UUnsmart No that you make

T

Telion•6/2/24, 3:50 AM

Wouldn't that mess with user's history a bit?

T

Telion•6/2/24, 3:51 AM

If I use any term llm will probably remember thatn and just assume that user either already knows about it or is still referring to it

T

Telion•6/2/24, 4:15 AM

Well, I tried, it got a lot more stable which is a huge improvement so far, although it still deviates a lot from what I get on meta.ai on the same model for some reason. On meta.ai i literally get perfect results. I am not sure what may be different there

We are considering migrating our existing Azure Whisper integration to use Cloudflare Whisper model

Similar Threads

Similar Threads

Similar Threads