The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model stops mid-sentence

The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model stops mid-sentence when prompt/messages + generated text goes above that limit.

RRaylight The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model...

Isaac McFadyen•5/17/24, 2:40 PM

Interesting - doesn't seem to be listed on the model page but the limit for LLaMA 2 is 2048 tokens so perhaps they share similar limits? https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8/#properties

Isaac McFadyen•5/17/24, 2:40 PM

The 4096 limit I found was for Mistral v0.2: https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.2/

Isaac McFadyen•5/17/24, 2:40 PM

So I expect each model is different.

Isaac McFadyen•5/17/24, 2:41 PM

Edited my original message to clarify, thanks for bringing it up

IIsaac McFadyen Interesting - doesn't seem to be listed on the model page but the limit for LLaM...

RaylightOP•5/17/24, 3:08 PM

No idea. I'm reconstructing the raw prompt and counting with the total with https://github.com/belladoreai/llama3-tokenizer-js. The limit is 2827 for some reason.

IIsaac McFadyen Note that depending on your source of WebM files (if it's the browser microphone...

scotto•5/17/24, 7:30 PM

sure that works on mobile?

Sscotto sure that works on mobile?

Isaac McFadyen•5/17/24, 7:40 PM

I'm not sure, no. You'd have to test.

Rraj.js How much is prompt limit for Llama 3 model? Is there any model on cloudfalre whi...

RaylightOP•5/17/24, 8:57 PM

Looks like

@cf/qwen/qwen1.5-7b-chat-awq

@cf/qwen/qwen1.5-7b-chat-awq

has a context length way above 8k. Managed to chop up and squeeze in ~20k tokens, ~100k characters, and ask a question about a detail buried in the beginning of the text.

SciCat•5/18/24, 8:23 PM

Hello I was wondering, do anyone know if it is possible to pass image as input ?

Sergey•5/20/24, 6:04 AM

Hey! Is there any rough ETA when text2img models will get out of beta?

SSciCat Hello I was wondering, do anyone know if it is possible to pass image as input ?

AngusMa•5/20/24, 2:06 PM

@cf/llava-hf/llava-1.5-7b-hf

@cf/llava-hf/llava-1.5-7b-hf

and

@cf/unum/uform-gen2-qwen-500m

@cf/unum/uform-gen2-qwen-500m

accept an image as input.
https://developers.cloudflare.com/workers-ai/models/#image-to-text

Cloudflare Docs

Models · Cloudflare Workers AI docs

Browse our entire catalog of models.

AngusMa•5/20/24, 2:11 PM

The model

bge-large-en-v1.5

bge-large-en-v1.5

appears on the model list, but not on the dashboard.
Was it forgotten?

AAngusMa The model `bge-large-en-v1.5` appears on the model list, but not on the dashboar...

RaylightOP•5/20/24, 2:31 PM

@cf/openchat/openchat-3.5-0106

@cf/openchat/openchat-3.5-0106

@cf/qwen/qwen1.5-14b-chat-awq

@cf/qwen/qwen1.5-14b-chat-awq

@hf/google/gemma-7b-it

@hf/google/gemma-7b-it

are also gone. Seems like the dashboard has a limit of 50 models.

Mmichelle taking a look - definitely not the intention.

Chaika•5/20/24, 6:10 PM

you've probably already figured it out, but fwiw the per_page query param on the search models endpoint has been broken for a while, I forgot to mention it and have just been working around it lol, it's always 50 no matter what you specify (in this case, the dash is trying 1000, but getting 50 back, and clicking through the list pages in the dash isn't paginating via the api)

Mrinank•5/20/24, 7:32 PM

can we use pinecone or upstash vector db in cf workers??

MMrinank can we use pinecone or upstash vector db in cf workers??

Mrinank•5/20/24, 7:35 PM

Getting an error while connecting : const pinecone = new Pinecone({ apiKey: '' });

[ERROR] Error compiling schema, function code: const schema2 = scope.schema[2];const schema1 = scope.schema[1]......

X [ERROR] Error in fetch handler: EvalError: Code generation from strings disallowed for this context

[ERROR] Error compiling schema, function code: const schema2 = scope.schema[2];const schema1 = scope.schema[1]......

X [ERROR] Error in fetch handler: EvalError: Code generation from strings disallowed for this context

MMrinank Getting an error while connecting : **const pinecone = new Pinecone({ apiKey: ''...

Isaac McFadyen•5/20/24, 8:50 PM

eval

eval

new Function

new Function

cannot be used in Workers for security reasons. Based on the error message I assume it's trying to do some sort of schema validation which often uses new Function.

Isaac McFadyen•5/20/24, 8:51 PM

If you have control over it then you can switch to a schema library that doesn't use new Function but if it's part of the pinecode library then that on't work.

Isaac McFadyen•5/20/24, 8:51 PM

You might be able to manually connect to Pinecone over HTTP if they have an HTTP API, or (more work) over TCP if needed.

Llui Hi! I'm evaluating the usage of Whisper from a cloudflare worker for the company...

catnaut.•5/21/24, 7:45 AM

encountered a similar issue. From my reading of the documentation, it seems that the only required input parameter is audio. However, I'm also interested in utilizing other parameters such as prompt, which are crucial for standardizing transcription format and providing hot words. Can anyone guide me on how to use these additional parameters with Whisper? Additionally, I'm looking for an OpenAI-compatible whisper endpoint, so I can migrate OpenAI services to Cloudflare. Any help would be greatly appreciated!

falex•5/23/24, 5:33 PM

I am working using '@hf/meta-llama/meta-llama-3-8b-instruct' in stream mode. the JSON answer has the below structure : data: {"response":" to","p":"abcdef"}
data: {"response":"Once","p":"abcdef"} . I not sure what's means the "p" parameter . any comments

falex•5/23/24, 5:34 PM

For example in this case "p" has a single letter data: {"response":" this","p":"a"}

Ffalex For example in this case "p" has a single letter data: {"response":" this","p":...

Chaika•5/23/24, 5:42 PM

https://blog.cloudflare.com/ai-side-channel-attack-mitigated

The Cloudflare Blog

Mitigating a token-length side-channel attack in our AI products

The Workers AI and AI Gateway team recently collaborated closely with security researchers at Ben Gurion University regarding a report submitted through our Public Bug Bounty program. Through this process, we discovered and fully patched a vulnerability affecting all LLM providers. Here’s the story

Chaika•5/23/24, 5:42 PM

it's a defense against that

is just random data

Nnoman i'm often getting these cancelled responses inside my worker with no info about ...

jason•5/24/24, 5:48 AM

@noman did you figure out why this is happening, I"m getting the same thing today with no reason as to why it might be happening?

POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16

POST http://workers-binding.ai/run?version=3 - Canceled @ 24/05/2024, 06:28:16

scotto•5/24/24, 5:24 PM

any news on the whisper webm format support? seems be reported several times on discord , quite importnat for web support

IIsaac McFadyen https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/MediaRecorder#mim...

Isaac McFadyen•5/24/24, 10:44 PM

@scotto have you tried this? I know you're targeting mobile and I'm not sure this will work on mobile, but it's definitely worth a shot since it might be a while still until WebM support is added natively to Workers AI.

Luca (steebchen)•5/26/24, 6:32 PM

Is there a way to use the AI gateway URL with AI workers in an OpenAI compatible way? It seems like workers AI has a different response and thus I can't just swap out my openai routes with CF AI when using AI gateway :/
there is this https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/ but it seems it doesn't work this way with AI gateway, because these URLs are fixed including the model name and do not expect /embeddings or /chat/completions

DISPLAY NAME•5/27/24, 2:52 AM

anyone tested finetune mistral with using collab t4 gpu? im not sure if it is not capable to do that and need to subscribe pro due to this error im having "subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'autotrain.trainers.clm', '--training_config', 'my-autotrain-llm/training_params.json']' died with <Signals.SIGKILL: 9>." or basically collab need more ram?

DISPLAY NAME•5/27/24, 2:53 AM

this is using autotrain huggingface suggested in documentation

LLiudmila In SDXL-lightning if I read the docs correctly you don't specify the width & hei...

RaylightOP•5/27/24, 12:17 PM

Afaik, Cloudflare is running the 1-step version so I'd expect it to be faster, but yeah.. width, height (as well as other parameters and features not yet but team said params are on the would be nice.

RaylightOP•5/27/24, 12:18 PM

On the topic of https://developers.cloudflare.com/workers-ai/models/stable-diffusion-xl-lightning/. Noticed that the model now returns (heavily) compressed JPGs instead of PNGs. Is this intentional?

RRaylight On the topic of <https://developers.cloudflare.com/workers-ai/models/stable-diff...

James•5/27/24, 3:33 PM

cc @michelle

whakamolenz•5/28/24, 12:45 AM

We are considering migrating our existing Azure Whisper integration to use Cloudflare Whisper model instead. We have both setup but finding Cloudflare whisper seems to be less accurate. Technically is there a difference in the model that could cause this?

Wwhakamolenz We are considering migrating our existing Azure Whisper integration to use Cloud...

Isaac McFadyen•5/28/24, 2:29 AM

Cloudflare's Whisper model is the basebase version of the model. Depending on what you're using on Azure it maybe be more accurate (I believe above base there is smallsmall, mediummedium, and largelarge, all of which will be more accurate)

IIsaac McFadyen Cloudflare's Whisper model is the `base` version of the model. Depending on what...

whakamolenz•5/28/24, 3:01 AM

ok thanks for the info

Mmichelle we are working to make it configurable per request

James•5/28/24, 4:16 PM

will it default back to png as documented?

Tempo•5/28/24, 8:05 PM

Hi, I have a question about links between AI and R2. I want to use a model for captioning images from R2, do I have to make a fetch in my worker code of the R2 image path before sending it to cf AI or is there a better way?
(example : https://developers.cloudflare.com/workers-ai/models/uform-gen2-qwen-500m/)

kNite•5/29/24, 12:19 AM

Hi, I have a problem with Cloudflare Workers + Hono + Cloudflare AI, I have a route defined (i.e. /v1/chat/completions) and when the user triggers that route I try to run env.AI.run([model], [etc]), but I get this error: TypeError: Cannot read properties of undefined (reading 'AI')TypeError: Cannot read properties of undefined (reading 'AI'). This is my Env interface:

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

interface Env {
    AI: any;
}

const env = (globalThis as any).env as Env

Also tried removing the const env = [...]const env = [...] and running AI inside a function with function exampleFunction(env: Env)function exampleFunction(env: Env) but it also didn't work.

liuxiaohua72•5/29/24, 12:39 AM

Is it possible for workers-ai to support OpenAI format API?

The limit for Llama 3 (on Cloudflare) seems to be ~2800 tokens though. The model stops mid-sentence

Similar Threads

Similar Threads

Similar Threads