As explained in the Workers AI docs for each model you want to use scoped prompts with hints (<https

As explained in the Workers AI docs for each model you want to use scoped prompts with hints (https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct/#scoped-prompts)

Basically instead of a single system message with the task and response type, you have the system message with the AIs goal followed by examples of how you want it to respond by using the user/assistant roles.

The example that CF specifically has in the docs:

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

{
  messages: [
    { role: "system", content: "you are a professional computer science assistant" },
    { role: "user", content: "what is WASM?" },
    { role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
    { role: "user", content: "does Python compile to WASM?" },
    { role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
    { role: "user", content: "what about Rust?" },
  ],
};

UUnsmart As explained in the Workers AI docs for each model you want to use scoped prompt...

Telion•6/2/24, 3:40 AM

Right, my issue was that scoped prompts don't improve the results, they actually make it worse. I specifically provided an example of scope prompts first, then with the images I demonstrated that without scoped prompts I get better results but they are still not any close to being what LLAMA on HF or Meta.ai provide.

Telion•6/2/24, 3:40 AM

The first example with scoped prompts I added above results in absolutely random results while putting everything into user prompts actually improves it by a lot

UnsmartOP•6/2/24, 3:42 AM

The example you showed technically has scoped prompts yes, but not used in a the best way. If you follow the prompting method recommended of

system: you do this
user: an example thing a user might say
assistant: what you want the response to be
user: example 2
assistant: answer 2
user: <actual input>

It will work wonderfully, I have had 0 issues with responses when prompting in that format.

Telion•6/2/24, 3:43 AM

Oh, I missed the assistant part. I noticed that not all models support this feature so I skipped it

Telion•6/2/24, 3:43 AM

In the example though assistant seems to be dynamic, does it get generated or how do I assign the right assistant prompt?

UnsmartOP•6/2/24, 3:44 AM

Its worked perfectly fine on every model I have used it on, no clue what doesnt support it... maybe the non text based ones

UnsmartOP•6/2/24, 3:45 AM

The assistant prompts are static, you just give it some generic examples. For a case where you want to have intent scoring you probably want to have like 2-3 examples for each category

Telion•6/2/24, 3:45 AM

Wait, is the assistant prompt just the history of answers?

Telion•6/2/24, 3:45 AM

that the llm made?

UnsmartOP•6/2/24, 3:45 AM

No that you make

UnsmartOP•6/2/24, 3:46 AM

You use the assistant to teach it how to respond

Telion•6/2/24, 3:46 AM

oh, wait I see

Telion•6/2/24, 3:46 AM

your example actually includes user prompts as part of the general mechanism describing it how ot behave

Telion•6/2/24, 3:46 AM

Let me try that, thank you!

Telion•6/2/24, 3:47 AM

I didn't think I can or should use user prompts to explain LLM something, I was 100% sure user prompts are jsut the user input and I shouldn't mess with that

UUnsmart No that you make

Telion•6/2/24, 3:50 AM

Wouldn't that mess with user's history a bit?

Telion•6/2/24, 3:51 AM

If I use any term llm will probably remember thatn and just assume that user either already knows about it or is still referring to it

Telion•6/2/24, 4:15 AM

Well, I tried, it got a lot more stable which is a huge improvement so far, although it still deviates a lot from what I get on meta.ai on the same model for some reason. On meta.ai i literally get perfect results. I am not sure what may be different there

Craig Dennis•6/3/24, 8:48 PM

Hi @Aurelios this might be just what you are looking for! https://github.com/craigsdennis/vanilla-chat-workers-ai/

GitHub

GitHub - craigsdennis/vanilla-chat-workers-ai

Contribute to craigsdennis/vanilla-chat-workers-ai development by creating an account on GitHub.

CCraig Dennis Hi @Aurelios this might be just what you are looking for! https://github.com/cra...

Victor•6/5/24, 4:54 PM

yoo, I'm gonna steal your json model list generation, but instead of beta/ga, I'll make it replace modelMappingsmodelMappings that's in @cloudflare/ai@cloudflare/ai but gone in

@cloudflare/workers-types

@cloudflare/workers-types

MR X•6/6/24, 3:19 AM

hello how to make cross x forwarded for ip in workers

VVictor yoo, I'm gonna steal your json model list generation, but instead of beta/ga, I'...

Craig Dennis•6/6/24, 11:14 PM

Steal away! And yeah modelMappings should be considered on it's way to deprecation

Atony•6/7/24, 2:59 AM

Hello, could you please ask if it is possible to integrate the Qwen2 series models into the Cloudflare AI platform? Thank you.

TurboFarmer9000•6/7/24, 2:31 PM

Was glad to see that cloudflare has whisper, but the particular model is not mentioned anywhere. I searched this channel and now i can see that it is the base model. OpenAI is $0.006 vs $0.0022 BUT it is running the large v3 model. If cloudflare adds more model options it would be attractive

FieraRyan•6/7/24, 10:50 PM

can I use beta llm model in prod? I need the latest model (llama 3-8b) but its not out of beta yet. Do you know the rate limit on free beta models?

FFieraRyan can I use beta llm model in prod? I need the latest model (llama 3-8b) but its n...

Victor•6/7/24, 11:25 PM

From a technical perspective, there's nothing stopping you. However beta comes without SLA guarantees and is simply "best effort". Usage is also free for the duration while in beta, but will be charged after. Also

Beta models may have lower rate limits while we work on performance and scale

https://developers.cloudflare.com/workers-ai/platform/limits/

Cloudflare Docs

Limits · Cloudflare Workers AI docs

Workers AI is now Generally Available. We’ve updated our rate limits to reflect this.

AAtony Hello, could you please ask if it is possible to integrate the Qwen2 series mode...

MissS•6/7/24, 11:45 PM

When qwen2 :O?

FieraRyan•6/8/24, 1:07 AM

Does this mean Paid Workers ($5) give 300000 neurons for free, or is this the upper limit?

UnsmartOP•6/8/24, 1:08 AM

its included usage, no upper limits beyond standard rate limits

MMissS When qwen2 :O?

Isaac McFadyen•6/8/24, 2:39 AM

Considering it was literally released yesterday, it might be a bit still

UUnsmart its included usage, no upper limits beyond standard rate limits

Fierylion•6/8/24, 1:48 PM

what does included mean? do you mean the $5 includes 300000 neurons?

rob•6/8/24, 7:53 PM

it's confusing

Rrob it's confusing

UnsmartOP•6/8/24, 8:19 PM

What is? The included usage? Not sure how else they could word it to say that you get usage included

Fierylion•6/8/24, 9:18 PM

are you sure its free 300000?

Fierylion•6/8/24, 9:19 PM

oh I see 10k a day, 30 days a month?

Fierylion•6/8/24, 9:19 PM

uh oh that means we get less than 10k a day for months with more than 30 days? xD

FFierylion uh oh that means we get less than 10k a day for months with more than 30 days? x...

Isaac McFadyen•6/9/24, 1:41 AM

You get 10k a day.

Isaac McFadyen•6/9/24, 1:41 AM

So in a month with 31 days, you get 310 000

jepcd•6/9/24, 4:23 AM

are there plans to add any multimodal models like clip?

I’d like to be able to store images and query them with text.

Using replicate rn, but would love to keep everything on cloudflare

Jjepcd are there plans to add any multimodal models like clip? I’d like to be able to ...

Fierylion•6/9/24, 4:08 PM

https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf/

Cloudflare Docs

llava-1.5-7b-hf · Cloudflare Workers AI docs

Run AI models in Workers, Pages, or via API.

Fierylion•6/9/24, 4:09 PM

llava 1.5 only support tiny tiny images (336x336) though

Fierylion•6/9/24, 4:09 PM

at least its cheap lol

Fierylion•6/9/24, 4:09 PM

I hope they add 1.6 which supports much bigger images

jepcd•6/9/24, 10:44 PM

yeah thats a bit too small unfortunately

Jjepcd yeah thats a bit too small unfortunately

Fierylion•6/10/24, 2:22 PM

to be fair CLIP models also only process tiny images 224X224 or 336x336

Afeelingmore•6/10/24, 11:53 PM

Hello everyone, I have a question regarding worker AI token management. I created a token through the REST API page, but I can’t seem to find where the tokens I’ve created are displayed. Can anyone help?

gauravmandall•6/11/24, 4:32 PM

I'm entering this command but somehow it's not working:

bunx wrangler vectorize create --dimensions=1536 ers-v1 --metric=cosine

bunx wrangler vectorize create --dimensions=1536 ers-v1 --metric=cosine

showing this error:

🚧 Creating index: 'ers-v1'

✘ [ERROR] A request to the Cloudflare API (/accounts/53e0c2270158721ff328e572f56950ea/vectorize/indexes) failed.

  vectorize.not_entitled [code: 1005]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

🚧 Creating index: 'ers-v1'

✘ [ERROR] A request to the Cloudflare API (/accounts/53e0c2270158721ff328e572f56950ea/vectorize/indexes) failed.

  vectorize.not_entitled [code: 1005]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose

#cloudflare-typescript #wrangler #vectorize-beta

As explained in the Workers AI docs for each model you want to use scoped prompts with hints (<https

Similar Threads