Hey @James Workers AI are very slow avg: 10sec Any tweaks we can do here? Thanks
Hey @James
Workers AI are very slow avg: 10sec
Any tweaks we can do here?
Thanks
Workers AI are very slow avg: 10sec
Any tweaks we can do here?
Thanks
Cif it's a text generation model you can lower the
max_tokensCyou'll get shorter replies though
SI managed the UI to stream the response
SThanks for the reply
RHow much is prompt limit for Llama 3 model? Is there any model on cloudfalre which has atleast 8k token limit
C4096 characters. no
I^ just to clarify this: that's tokens, not characters.
IA token is almost always more than 1 character (to see this for yourself, you can put your text in here, although note that this is not the same tokenizer that the Workers AI models use: https://platform.openai.com/tokenizer)
IBut yes, 4096 tokens maximum (for Mistral v0.2).
ADoes anyone know where I can find a price for Llama3-8b-instruct? There's no pricing for it on the models page, and it's not in the dropdown on the calculator.
https://developers.cloudflare.com/workers-ai/platform/pricing/#text-generation
https://ai.cloudflare.com/#pricing-calculator
https://developers.cloudflare.com/workers-ai/platform/pricing/#text-generation
https://ai.cloudflare.com/#pricing-calculator
Ihttps://developers.cloudflare.com/workers-ai/platform/pricing/ It's still beta so unbilled currently.
I>You can still enjoy unlimited usage on the beta models in the catalog until they graduate out of beta.
AThat's great news, I thought I was going to bre in for a huge bill...
Mdoes langsmith works with workers AI?
Snews?
INote that depending on your source of WebM files (if it's the browser microphone) you can take a look at that link I sent to force the browser to output a more standardized format
RThe limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model stops mid-sentence when prompt/messages + generated text goes above that limit.
IInteresting - doesn't seem to be listed on the model page but the limit for LLaMA 2 is 2048 tokens so perhaps they share similar limits? https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8/#properties
IThe 4096 limit I found was for Mistral v0.2: https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.2/
ISo I expect each model is different.
IEdited my original message to clarify, thanks for bringing it up 
RNo idea. I'm reconstructing the raw prompt and counting with the total with https://github.com/belladoreai/llama3-tokenizer-js. The limit is 2827 for some reason.
Ssure that works on mobile?
II'm not sure, no. You'd have to test.
RLooks like
@cf/qwen/qwen1.5-7b-chat-awq has a context length way above 8k. Managed to chop up and squeeze in ~20k tokens, ~100k characters, and ask a question about a detail buried in the beginning of the text.
SHello I was wondering, do anyone know if it is possible to pass image as input ?
SHey! Is there any rough ETA when text2img models will get out of beta?
A@cf/llava-hf/llava-1.5-7b-hf and @cf/unum/uform-gen2-qwen-500m accept an image as input.https://developers.cloudflare.com/workers-ai/models/#image-to-text
AThe model
Was it forgotten?
bge-large-en-v1.5 appears on the model list, but not on the dashboard.Was it forgotten?
R@cf/openchat/openchat-3.5-0106, @cf/qwen/qwen1.5-14b-chat-awq, @hf/google/gemma-7b-it are also gone. Seems like the dashboard has a limit of 50 models.
Cyou've probably already figured it out, but fwiw the per_page query param on the search models endpoint has been broken for a while, I forgot to mention it and have just been working around it lol, it's always 50 no matter what you specify (in this case, the dash is trying 1000, but getting 50 back, and clicking through the list pages in the dash isn't paginating via the api)
Mcan we use pinecone or upstash vector db in cf workers??
MGetting an error while connecting : const pinecone = new Pinecone({ apiKey: '' });
Ieval or new Function cannot be used in Workers for security reasons. Based on the error message I assume it's trying to do some sort of schema validation which often uses new Function.IIf you have control over it then you can switch to a schema library that doesn't use new Function but if it's part of the pinecode library then that on't work.
IYou might be able to manually connect to Pinecone over HTTP if they have an HTTP API, or (more work) over TCP if needed.
Cencountered a similar issue. From my reading of the documentation, it seems that the only required input parameter is audio. However, I'm also interested in utilizing other parameters such as prompt, which are crucial for standardizing transcription format and providing hot words. Can anyone guide me on how to use these additional parameters with Whisper? Additionally, I'm looking for an OpenAI-compatible whisper endpoint, so I can migrate OpenAI services to Cloudflare. Any help would be greatly appreciated!
FI am working using '@hf/meta-llama/meta-llama-3-8b-instruct' in stream mode. the JSON answer has the below structure : data: {"response":" to","p":"abcdef"}
data: {"response":"Once","p":"abcdef"} . I not sure what's means the "p" parameter . any comments
data: {"response":"Once","p":"abcdef"} . I not sure what's means the "p" parameter . any comments
FFor example in this case "p" has a single letter data: {"response":" this","p":"a"}
CThe Cloudflare Blog
The Workers AI and AI Gateway team recently collaborated closely with security researchers at Ben Gurion University regarding a report submitted through our Public Bug Bounty program. Through this process, we discovered and fully patched a vulnerability affecting all LLM providers. Here’s the story
Cit's a defense against that
p is just random data
J@noman did you figure out why this is happening, I"m getting the same thing today with no reason as to why it might be happening?
POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16