For example in this case "p" has a single letter data: {"response":" this","p":"a"}

Ffalex For example in this case "p" has a single letter data: {"response":" this","p":...

C

Chaika•5/23/24, 5:42 PM

https://blog.cloudflare.com/ai-side-channel-attack-mitigated

The Cloudflare Blog

Mitigating a token-length side-channel attack in our AI products

The Workers AI and AI Gateway team recently collaborated closely with security researchers at Ben Gurion University regarding a report submitted through our Public Bug Bounty program. Through this process, we discovered and fully patched a vulnerability affecting all LLM providers. Here’s the story

C

Chaika•5/23/24, 5:42 PM

it's a defense against that

is just random data

Nnoman i'm often getting these cancelled responses inside my worker with no info about ...

J

jason•5/24/24, 5:48 AM

@noman did you figure out why this is happening, I"m getting the same thing today with no reason as to why it might be happening?

POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16

POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16

POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16

POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16

S

scotto•5/24/24, 5:24 PM

any news on the whisper webm format support? seems be reported several times on discord , quite importnat for web support

IIsaac McFadyen https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/MediaRecorder#mim...

I

Isaac McFadyen•5/24/24, 10:44 PM

@scotto have you tried this? I know you're targeting mobile and I'm not sure this will work on mobile, but it's definitely worth a shot since it might be a while still until WebM support is added natively to Workers AI.

L

Luca (steebchen)•5/26/24, 6:32 PM

Is there a way to use the AI gateway URL with AI workers in an OpenAI compatible way? It seems like workers AI has a different response and thus I can't just swap out my openai routes with CF AI when using AI gateway :/
there is this https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/ but it seems it doesn't work this way with AI gateway, because these URLs are fixed including the model name and do not expect /embeddings or /chat/completions

D

DISPLAY NAME•5/27/24, 2:52 AM

anyone tested finetune mistral with using collab t4 gpu? im not sure if it is not capable to do that and need to subscribe pro due to this error im having "subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'autotrain.trainers.clm', '--training_config', 'my-autotrain-llm/training_params.json']' died with <Signals.SIGKILL: 9>." or basically collab need more ram?

D

DISPLAY NAME•5/27/24, 2:53 AM

this is using autotrain huggingface suggested in documentation

LLiudmila In SDXL-lightning if I read the docs correctly you don't specify the width & hei...

R

Raylight•5/27/24, 12:17 PM

Afaik, Cloudflare is running the 1-step version so I'd expect it to be faster, but yeah.. width, height (as well as other parameters and features not yet but team said params are on the would be nice.

R

Raylight•5/27/24, 12:18 PM

On the topic of https://developers.cloudflare.com/workers-ai/models/stable-diffusion-xl-lightning/. Noticed that the model now returns (heavily) compressed JPGs instead of PNGs. Is this intentional?

RRaylight On the topic of <https://developers.cloudflare.com/workers-ai/models/stable-diff...

J

James•5/27/24, 3:33 PM

cc @michelle

W

whakamolenz•5/28/24, 12:45 AM

We are considering migrating our existing Azure Whisper integration to use Cloudflare Whisper model instead. We have both setup but finding Cloudflare whisper seems to be less accurate. Technically is there a difference in the model that could cause this?

Wwhakamolenz We are considering migrating our existing Azure Whisper integration to use Cloud...

I

Isaac McFadyen•5/28/24, 2:29 AM

Cloudflare's Whisper model is the

base

base

version of the model. Depending on what you're using on Azure it maybe be more accurate (I believe above base there is

small

small

small

small,

medium

medium

medium

medium, and

large

large

large

large, all of which will be more accurate)

IIsaac McFadyen Cloudflare's Whisper model is the `base` version of the model. Depending on what...

W

whakamolenz•5/28/24, 3:01 AM

ok thanks for the info

Mmichelle we are working to make it configurable per request

J

James•5/28/24, 4:16 PM

will it default back to png as documented?

T

Tempo•5/28/24, 8:05 PM

Hi, I have a question about links between AI and R2. I want to use a model for captioning images from R2, do I have to make a fetch in my worker code of the R2 image path before sending it to cf AI or is there a better way?
(example : https://developers.cloudflare.com/workers-ai/models/uform-gen2-qwen-500m/)

K

kNite•5/29/24, 12:19 AM

Hi, I have a problem with Cloudflare Workers + Hono + Cloudflare AI, I have a route defined (i.e. /v1/chat/completions) and when the user triggers that route I try to run env.AI.run([model], [etc]), but I get this error:

TypeError: Cannot read properties of undefined (reading 'AI')

TypeError: Cannot read properties of undefined (reading 'AI')

TypeError: Cannot read properties of undefined (reading 'AI')

TypeError: Cannot read properties of undefined (reading 'AI'). This is my Env interface:

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

Also tried removing the

const env = [...]

const env = [...]

const env = [...]

const env = [...] and running AI inside a function with

function exampleFunction(env: Env)

function exampleFunction(env: Env)

function exampleFunction(env: Env)

function exampleFunction(env: Env) but it also didn't work.

L

liuxiaohua72•5/29/24, 12:39 AM

Is it possible for workers-ai to support OpenAI format API?

JJames will it default back to png as documented?

J

James•5/30/24, 3:59 PM

any updates here @michelle ? Is this an expected breaking change, or will this be reverted back to png?

N

Nicolas92•5/30/24, 8:14 PM

Hi guys, how can use this worker ai ? Have someone the Java script for it ?

NNicolas92 Hi guys, how can use this worker ai ? Have someone the Java script for it ?

I

Isaac McFadyen•5/30/24, 8:58 PM

You can follow any of these guides. Use CLI or Dashboard if you want to deploy a Worker, use the API one for if you want to access it externally without a Worker: https://developers.cloudflare.com/workers-ai/get-started/

J

js•5/31/24, 5:34 AM

Will llama 3 loras be supported soon? https://developers.cloudflare.com/workers-ai/fine-tunes/loras/

Cloudflare Docs

Using LoRA adapters · Cloudflare Workers AI docs

Upload and use LoRA adapters to get fine-tuned inference on Workers AI.

P

Puliczek•6/1/24, 2:07 PM

Hi, are there any startups that are using "Workers for Platforms"?

PPuliczek Hi, are there any startups that are using "Workers for Platforms"?

I

Isaac McFadyen•6/1/24, 5:04 PM

This would probably be a better question for #workers-for-platforms. This channel is for Workers AI.

Mmichelle sorry - was at Connect! We are working on making it configurable per request, TB...

R

Raylight•6/1/24, 6:58 PM

Hmm.. quite a lot going on with that model right now. Was going to give feedback on the compression level as the immediate issue, but now the model is broken for deployed workers that are using SDK 1.0.53 or earlier. Which is kinda.. more immediate. Feels like the model is actively being worked on right in the open, so I'm thinking.. should I just hold off a bit, or try to play catch up with feedback?

Mmichelle are you using native binding? or the npm install?

R

Raylight•6/1/24, 7:05 PM

I'm using all of them. 1.0.53, 1.1.0, native binding, api.

T

Telion•6/1/24, 7:28 PM

Hi, I was wondering whether there is any way to improve the results of the LLAMA 3 here. I've been trying

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct and

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct and both provide me with almost completely random responses when trying to use a system message along the prompt.

I am trying to classify user prompts by a list of intents. An example of prompt I am currently struggling with is "What would I need to do to learn js?". While example of intents is:
'1 Track progress and view improvements',
'2 Generate a learning plan or prepare learning materials list',
'3 Need an explanation',
'4 Seek motivation',

Here's how I prompt the model with the data:

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

The responses to this are pretty random. I get:
1
4
3
2
3

When I moved the message from the system prompt into user prompt it started working better but the results are still really unreliable. I wanted to check with experienced people here whether my prompting is too unprecise or there is something wrong with how I understand the CF's AI.

The real issue here is that when I just use the same model on HF or Meta or even drop it into gemini or gpt-4o/3.5t all of them return perfect results, straight "2"s but not on CF

T

Telion•6/1/24, 7:30 PM

I've used the same prompt on all of those, the results differ majorly with what I experience on CF's

T

Telion•6/1/24, 7:32 PM

Although when I put the same message into user's prompt and get rid of the system prompt it starts working a lot better. Yet other prompts still fail randomly, while seem to be percieved correctly on other platforms.

system-prompt-seems-to-not-enforce-behavior-v0-z6f8jse1fz3d1.webp

system-prompt-seems-to-not-enforce-behavior-v0-g6geobw7fz3d1.webp

TTelion Hi, I was wondering whether there is any way to improve the results of the LLAMA...

U

Unsmart•6/2/24, 2:41 AM

As explained in the Workers AI docs for each model you want to use scoped prompts with hints (https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct/#scoped-prompts)

Basically instead of a single system message with the task and response type, you have the system message with the AIs goal followed by examples of how you want it to respond by using the user/assistant roles.

The example that CF specifically has in the docs:

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

UUnsmart As explained in the Workers AI docs for each model you want to use scoped prompt...

T

Telion•6/2/24, 3:40 AM

Right, my issue was that scoped prompts don't improve the results, they actually make it worse. I specifically provided an example of scope prompts first, then with the images I demonstrated that without scoped prompts I get better results but they are still not any close to being what LLAMA on HF or Meta.ai provide.

T

Telion•6/2/24, 3:40 AM

The first example with scoped prompts I added above results in absolutely random results while putting everything into user prompts actually improves it by a lot

U

Unsmart•6/2/24, 3:42 AM

The example you showed technically has scoped prompts yes, but not used in a the best way. If you follow the prompting method recommended of

system: you do this
user: an example thing a user might say
assistant: what you want the response to be
user: example 2
assistant: answer 2
user: <actual input>

It will work wonderfully, I have had 0 issues with responses when prompting in that format.

T

Telion•6/2/24, 3:43 AM

Oh, I missed the assistant part. I noticed that not all models support this feature so I skipped it

T

Telion•6/2/24, 3:43 AM

In the example though assistant seems to be dynamic, does it get generated or how do I assign the right assistant prompt?

For example in this case "p" has a single letter data: {"response":" this","p":"a"}

Similar Threads