Hi, I was wondering whether there is any way to improve the results of the LLAMA 3 here. I've been t

Hi, I was wondering whether there is any way to improve the results of the LLAMA 3 here. I've been trying

@/hf/meta-llama/meta-llama-3-8b-instruct

@/hf/meta-llama/meta-llama-3-8b-instruct

and

@cf/meta/llama-3-8b-instruct

@cf/meta/llama-3-8b-instruct

and both provide me with almost completely random responses when trying to use a system message along the prompt.

I am trying to classify user prompts by a list of intents. An example of prompt I am currently struggling with is "What would I need to do to learn js?". While example of intents is:
'1 Track progress and view improvements',
'2 Generate a learning plan or prepare learning materials list',
'3 Need an explanation',
'4 Seek motivation',

Here's how I prompt the model with the data:

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

        const intent = await env.AI.run('@cf/meta/llama-3-8b-instruct', {
          messages: [
            {
              "role": "system",
              "content": `I will help you figure out intent of the message based on available options. Available intents:\n${INTENTS.join('\n')}\n\nI will choose an intent from the list that best matches the user's prompt and return only its number. There will be no other text in my response other then the found number. I will use the following format: [intent number from the list].\n\nExample of response when found:\n3\n\nExample of when intent cannot be determined:\nNo intent`
            },
            {
              "role": "user",
              "content": prompt
            }
          ]
        });

The responses to this are pretty random. I get:
1
4
3
2
3

When I moved the message from the system prompt into user prompt it started working better but the results are still really unreliable. I wanted to check with experienced people here whether my prompting is too unprecise or there is something wrong with how I understand the CF's AI.

The real issue here is that when I just use the same model on HF or Meta or even drop it into gemini or gpt-4o/3.5t all of them return perfect results, straight "2"s but not on CF

TelionOP•6/1/24, 7:30 PM

I've used the same prompt on all of those, the results differ majorly with what I experience on CF's

TelionOP•6/1/24, 7:32 PM

Although when I put the same message into user's prompt and get rid of the system prompt it starts working a lot better. Yet other prompts still fail randomly, while seem to be percieved correctly on other platforms.

system-prompt-seems-to-not-enforce-behavior-v0-z6f8jse1fz3d1.webp

system-prompt-seems-to-not-enforce-behavior-v0-g6geobw7fz3d1.webp

TTelion Hi, I was wondering whether there is any way to improve the results of the LLAMA...

Unsmart•6/2/24, 2:41 AM

As explained in the Workers AI docs for each model you want to use scoped prompts with hints (https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct/#scoped-prompts)

Basically instead of a single system message with the task and response type, you have the system message with the AIs goal followed by examples of how you want it to respond by using the user/assistant roles.

The example that CF specifically has in the docs:

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

UUnsmart As explained in the Workers AI docs for each model you want to use scoped prompt...

TelionOP•6/2/24, 3:40 AM

Right, my issue was that scoped prompts don't improve the results, they actually make it worse. I specifically provided an example of scope prompts first, then with the images I demonstrated that without scoped prompts I get better results but they are still not any close to being what LLAMA on HF or Meta.ai provide.

TelionOP•6/2/24, 3:40 AM

The first example with scoped prompts I added above results in absolutely random results while putting everything into user prompts actually improves it by a lot

Unsmart•6/2/24, 3:42 AM

The example you showed technically has scoped prompts yes, but not used in a the best way. If you follow the prompting method recommended of

system: you do this
user: an example thing a user might say
assistant: what you want the response to be
user: example 2
assistant: answer 2
user: <actual input>

It will work wonderfully, I have had 0 issues with responses when prompting in that format.

TelionOP•6/2/24, 3:43 AM

Oh, I missed the assistant part. I noticed that not all models support this feature so I skipped it

TelionOP•6/2/24, 3:43 AM

In the example though assistant seems to be dynamic, does it get generated or how do I assign the right assistant prompt?

Unsmart•6/2/24, 3:44 AM

Its worked perfectly fine on every model I have used it on, no clue what doesnt support it... maybe the non text based ones

Unsmart•6/2/24, 3:45 AM

The assistant prompts are static, you just give it some generic examples. For a case where you want to have intent scoring you probably want to have like 2-3 examples for each category

TelionOP•6/2/24, 3:45 AM

Wait, is the assistant prompt just the history of answers?

TelionOP•6/2/24, 3:45 AM

that the llm made?

Unsmart•6/2/24, 3:45 AM

No that you make

Unsmart•6/2/24, 3:46 AM

You use the assistant to teach it how to respond

TelionOP•6/2/24, 3:46 AM

oh, wait I see

TelionOP•6/2/24, 3:46 AM

your example actually includes user prompts as part of the general mechanism describing it how ot behave

TelionOP•6/2/24, 3:46 AM

Let me try that, thank you!

TelionOP•6/2/24, 3:47 AM

I didn't think I can or should use user prompts to explain LLM something, I was 100% sure user prompts are jsut the user input and I shouldn't mess with that

UUnsmart No that you make

TelionOP•6/2/24, 3:50 AM

Wouldn't that mess with user's history a bit?

TelionOP•6/2/24, 3:51 AM

If I use any term llm will probably remember thatn and just assume that user either already knows about it or is still referring to it

TelionOP•6/2/24, 4:15 AM

Well, I tried, it got a lot more stable which is a huge improvement so far, although it still deviates a lot from what I get on meta.ai on the same model for some reason. On meta.ai i literally get perfect results. I am not sure what may be different there

Craig Dennis•6/3/24, 8:48 PM

Hi @Aurelios this might be just what you are looking for! https://github.com/craigsdennis/vanilla-chat-workers-ai/

GitHub

GitHub - craigsdennis/vanilla-chat-workers-ai

Contribute to craigsdennis/vanilla-chat-workers-ai development by creating an account on GitHub.

CCraig Dennis Hi @Aurelios this might be just what you are looking for! https://github.com/cra...

Victor•6/5/24, 4:54 PM

yoo, I'm gonna steal your json model list generation, but instead of beta/ga, I'll make it replace

modelMappings

modelMappings

that's in

@cloudflare/ai

@cloudflare/ai

but gone in

@cloudflare/workers-types

@cloudflare/workers-types

MR X•6/6/24, 3:19 AM

hello how to make cross x forwarded for ip in workers

VVictor yoo, I'm gonna steal your json model list generation, but instead of beta/ga, I'...

Craig Dennis•6/6/24, 11:14 PM

Steal away! And yeah modelMappings should be considered on it's way to deprecation

Atony•6/7/24, 2:59 AM

Hello, could you please ask if it is possible to integrate the Qwen2 series models into the Cloudflare AI platform? Thank you.

TurboFarmer9000•6/7/24, 2:31 PM

Was glad to see that cloudflare has whisper, but the particular model is not mentioned anywhere. I searched this channel and now i can see that it is the base model. OpenAI is $0.006 vs $0.0022 BUT it is running the large v3 model. If cloudflare adds more model options it would be attractive

FieraRyan•6/7/24, 10:50 PM

can I use beta llm model in prod? I need the latest model (llama 3-8b) but its not out of beta yet. Do you know the rate limit on free beta models?

FFieraRyan can I use beta llm model in prod? I need the latest model (llama 3-8b) but its n...

Victor•6/7/24, 11:25 PM

From a technical perspective, there's nothing stopping you. However beta comes without SLA guarantees and is simply "best effort". Usage is also free for the duration while in beta, but will be charged after. Also

Beta models may have lower rate limits while we work on performance and scale

https://developers.cloudflare.com/workers-ai/platform/limits/

Cloudflare Docs

Limits · Cloudflare Workers AI docs

Workers AI is now Generally Available. We’ve updated our rate limits to reflect this.

AAtony Hello, could you please ask if it is possible to integrate the Qwen2 series mode...

MissS•6/7/24, 11:45 PM

When qwen2 :O?

FieraRyan•6/8/24, 1:07 AM

Does this mean Paid Workers ($5) give 300000 neurons for free, or is this the upper limit?

Unsmart•6/8/24, 1:08 AM

its included usage, no upper limits beyond standard rate limits

MMissS When qwen2 :O?

Isaac McFadyen•6/8/24, 2:39 AM

Considering it was literally released yesterday, it might be a bit still

UUnsmart its included usage, no upper limits beyond standard rate limits

Fierylion•6/8/24, 1:48 PM

what does included mean? do you mean the $5 includes 300000 neurons?

rob•6/8/24, 7:53 PM

it's confusing

Rrob it's confusing

Unsmart•6/8/24, 8:19 PM

What is? The included usage? Not sure how else they could word it to say that you get usage included

Fierylion•6/8/24, 9:18 PM

are you sure its free 300000?

Fierylion•6/8/24, 9:19 PM

oh I see 10k a day, 30 days a month?

Fierylion•6/8/24, 9:19 PM

uh oh that means we get less than 10k a day for months with more than 30 days? xD

FFierylion uh oh that means we get less than 10k a day for months with more than 30 days? x...

Isaac McFadyen•6/9/24, 1:41 AM

You get 10k a day.

Isaac McFadyen•6/9/24, 1:41 AM

So in a month with 31 days, you get 310 000

jepcd•6/9/24, 4:23 AM

are there plans to add any multimodal models like clip?

I’d like to be able to store images and query them with text.

Using replicate rn, but would love to keep everything on cloudflare

Jjepcd are there plans to add any multimodal models like clip? I’d like to be able to ...

Fierylion•6/9/24, 4:08 PM

https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/

Cloudflare Docs

llava-1.5-7b-hf · Cloudflare Workers AI docs

Run AI models in Workers, Pages, or via API.

Fierylion•6/9/24, 4:09 PM

llava 1.5 only support tiny tiny images (336x336) though

Fierylion•6/9/24, 4:09 PM

at least its cheap lol

Fierylion•6/9/24, 4:09 PM

I hope they add 1.6 which supports much bigger images

jepcd•6/9/24, 10:44 PM

yeah thats a bit too small unfortunately

Hi, I was wondering whether there is any way to improve the results of the LLAMA 3 here. I've been t

Similar Threads