Hello, do you plan to add Stable Diffusion 3 Medium to Model Catalog?

L

Luka•6/13/24, 11:44 AM

Any plans to add newer OS embedding models?

L

Luka•6/13/24, 11:44 AM

I'm building a speedy search where my API is in the worker and using bge small seems much nicer. But it has falled behind in constrast to the latest & greatest...

FFerdi KIZILTOPRAK Hello, do you plan to add Stable Diffusion 3 Medium to Model Catalog?

I

Isaac McFadyen•6/13/24, 1:40 PM

Given that Stability AI has an extremely restrictive commercial license, and that the model actually isn't that good compared to SDXL (especially the fine-tuned SDXL models) I highly doubt it.

I

icyfox•6/17/24, 1:25 PM

Are there plans to add constrained generation (specifically json output) to the LLMs and LORA hosting of models on Workers?

I know there was some conversation about this back in March, but haven't seen any updates since: workers-ai

R

rob•6/17/24, 7:06 PM

is there a list of text models on workers AI support json mode , if any?

Rrob is there a list of text models on workers AI support json mode , if any?

I

Isaac McFadyen•6/17/24, 8:27 PM

None currently support JSON-constrained outputs (although you can tell it to output JSON and try and parse it yourself). The person right above your message was asking for updates as well but as far as I know there are no updates.

R

rob•6/17/24, 8:43 PM

I went thru them all, looks like @hf/nousresearch/hermes-2-pro-mistral-7b suports json mode @icyfox

R

rob•6/17/24, 8:44 PM

https://developers.cloudflare.com/workers-ai/models/hermes-2-pro-mistral-7b/ at least thats what the docs said.

But im working on something now that should be able to get json back from most of the cf ai text models, still testing it

Cloudflare Docs

hermes-2-pro-mistral-7b · Cloudflare Workers AI docs

Run AI models in Workers, Pages, or via API.

Rrob I went thru them all, looks like @hf/nousresearch/hermes-2-pro-mistral-7b suport...

I

Isaac McFadyen•6/17/24, 9:02 PM

The model itself supports JSON mode, but it's not the same as JSON constraining. The model should output JSON if you prompt it properly, but it's not guaranteed like proper JSON constraining is.

R

rob•6/18/24, 12:36 AM

yeah makes sense

T

Tojak•6/18/24, 3:40 PM

Hi, how can I get statistics of workers AI usage? I don't see any api documentation for it? Is it possible?

IIsaac McFadyen None currently support JSON-constrained outputs (although you can tell it to out...

I

icyfox•6/18/24, 4:21 PM

Thanks @Isaac McFadyen | YYZ01, EWR01, that's what I assumed too based on the docs. Looks like it's not a good fit for my usecase (lora r limitations, lack of constrained schemas) but will follow along in case any updates land soon.

A

Ali_hat•6/19/24, 3:36 PM

Hey everyone, I'm currently using

llama-3-8b-instruct

llama-3-8b-instruct

with the REST API. It works somewhat but it sometimes feels like it's trying to chat, remembering context from previous HTTP calls. Has anyone ran into similar issues? I've been refining the prompts using system, user and assistant roles.

R

rob•6/19/24, 3:46 PM

@Ali_hat what type of tasks are you trying to get from it ? or more so what's the use case

Iicyfox Thanks @Isaac McFadyen | YYZ01, EWR01, that's what I assumed too based on the do...

R

rob•6/19/24, 3:48 PM

I have something I made for this exctly, for getting models to return your structured data properly DM me if you want a link

I

icyfox•6/19/24, 6:25 PM

@rob Without logit-level constraints it's not possible to have a guarantee that it will return structured data every time.

R

rob•6/19/24, 6:26 PM

oh yea for sure. it'll either retry and error but it's a drastic increase over other available methods or having to prompt engineer every single model

Rrob @Ali_hat what type of tasks are you trying to get from it ? or more so what's t...

A

Ali_hat•6/19/24, 9:35 PM

Hi @rob, I basically use it for Q&A based on customer data. I don't need structured data as I only need plain text. Here's the code in a nutshell.

const systemContent = `You are a knowledgeable employee familiar with the company ${companyName}, responding to customer inquiries. Follow these guidelines:
    - Answer in the same language as the question.
    - Do not reveal your identity.
    - If you don't know the answer, admit it without making anything up.
    - Maintain a neutral tone.
    - Do not provide opinions or personal views.
    - Avoid asking for feedback.
    - Keep the conversation strictly to the point; do not engage in small talk or recommendations.
    - Do not apologize.
    - Do not initiate or continue small talk.
    - Do not use phrases like "I'm sorry" or "I apologize."`;
await got.post(`https://api.cloudflare.com/client/v4/accounts/${Env.CLOUDFLARE_ACCOUNT_ID}/ai/run/${model}`, {
      headers: { Authorization: `Bearer ${Env.CLOUDFLARE_WORKERS_AI_KEY}` },
      json: {
        max_tokens: 350,
        messages: [
          { role: 'system', content: systemContent },
          { role: 'user', content: `Question:${question}` },
          { role: 'assistant', content: context }
        ],
        temperature: 0.5
      }
    });

const systemContent = `You are a knowledgeable employee familiar with the company ${companyName}, responding to customer inquiries. Follow these guidelines:
    - Answer in the same language as the question.
    - Do not reveal your identity.
    - If you don't know the answer, admit it without making anything up.
    - Maintain a neutral tone.
    - Do not provide opinions or personal views.
    - Avoid asking for feedback.
    - Keep the conversation strictly to the point; do not engage in small talk or recommendations.
    - Do not apologize.
    - Do not initiate or continue small talk.
    - Do not use phrases like "I'm sorry" or "I apologize."`;
await got.post(`https://api.cloudflare.com/client/v4/accounts/${Env.CLOUDFLARE_ACCOUNT_ID}/ai/run/${model}`, {
      headers: { Authorization: `Bearer ${Env.CLOUDFLARE_WORKERS_AI_KEY}` },
      json: {
        max_tokens: 350,
        messages: [
          { role: 'system', content: systemContent },
          { role: 'user', content: `Question:${question}` },
          { role: 'assistant', content: context }
        ],
        temperature: 0.5
      }
    });

const systemContent = `You are a knowledgeable employee familiar with the company ${companyName}, responding to customer inquiries. Follow these guidelines:
    - Answer in the same language as the question.
    - Do not reveal your identity.
    - If you don't know the answer, admit it without making anything up.
    - Maintain a neutral tone.
    - Do not provide opinions or personal views.
    - Avoid asking for feedback.
    - Keep the conversation strictly to the point; do not engage in small talk or recommendations.
    - Do not apologize.
    - Do not initiate or continue small talk.
    - Do not use phrases like "I'm sorry" or "I apologize."`;
await got.post(`https://api.cloudflare.com/client/v4/accounts/${Env.CLOUDFLARE_ACCOUNT_ID}/ai/run/${model}`, {
      headers: { Authorization: `Bearer ${Env.CLOUDFLARE_WORKERS_AI_KEY}` },
      json: {
        max_tokens: 350,
        messages: [
          { role: 'system', content: systemContent },
          { role: 'user', content: `Question:${question}` },
          { role: 'assistant', content: context }
        ],
        temperature: 0.5
      }
    });

const systemContent = `You are a knowledgeable employee familiar with the company ${companyName}, responding to customer inquiries. Follow these guidelines:
    - Answer in the same language as the question.
    - Do not reveal your identity.
    - If you don't know the answer, admit it without making anything up.
    - Maintain a neutral tone.
    - Do not provide opinions or personal views.
    - Avoid asking for feedback.
    - Keep the conversation strictly to the point; do not engage in small talk or recommendations.
    - Do not apologize.
    - Do not initiate or continue small talk.
    - Do not use phrases like "I'm sorry" or "I apologize."`;
await got.post(`https://api.cloudflare.com/client/v4/accounts/${Env.CLOUDFLARE_ACCOUNT_ID}/ai/run/${model}`, {
      headers: { Authorization: `Bearer ${Env.CLOUDFLARE_WORKERS_AI_KEY}` },
      json: {
        max_tokens: 350,
        messages: [
          { role: 'system', content: systemContent },
          { role: 'user', content: `Question:${question}` },
          { role: 'assistant', content: context }
        ],
        temperature: 0.5
      }
    });

R

rob•6/19/24, 9:43 PM

remembering context from previous HTTP calls

what do you mean there?

Rrob remembering context from previous HTTP calls what do you mean there?

℠

℠•6/20/24, 3:13 AM

I’d bet my bald head that he means the model keeps training on chat-data instead of starting every chat-session fresh (from base model) .

I

Isaac McFadyen•6/20/24, 3:20 AM

Which is not the case - the model only remembers history/messages you give it (if anything) and is not automatically trained or tuned over time.

H

hydroxide•6/20/24, 6:43 AM

has anyone successfully used their own fine-tuned lora? I can't successfully upload a

gemma-7b-it

gemma-7b-it

gemma-7b-it

gemma-7b-it finetune at all (

'file' should be of valid safetensors type

'file' should be of valid safetensors type

'file' should be of valid safetensors type

'file' should be of valid safetensors type), and I can successfully upload/run inference for

mistral/mistral-7b-instruct-v0.2-lora

mistral/mistral-7b-instruct-v0.2-lora

mistral/mistral-7b-instruct-v0.2-lora

mistral/mistral-7b-instruct-v0.2-lora, but the results are worse than the vanilla model

H

hydroxide•6/20/24, 6:43 AM

the lora works fine outside cloudflare

H

hydroxide•6/20/24, 6:44 AM

and per other comments, I've restricted the modules to

q_proj

q_proj

q_proj

q_proj and

v_proj

v_proj

v_proj

v_proj

H

hydroxide•6/20/24, 6:45 AM

I finetuned

mistralai/mistral-7b-instruct-v0.2

mistralai/mistral-7b-instruct-v0.2

mistralai/mistral-7b-instruct-v0.2

mistralai/mistral-7b-instruct-v0.2 using the

AutoTrain_LLM

AutoTrain_LLM

AutoTrain_LLM

AutoTrain_LLM notebook , then cURL to

https://api.cloudflare.com/client/v4/accounts/${account_id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora

https://api.cloudflare.com/client/v4/accounts/${account_id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora

https://api.cloudflare.com/client/v4/accounts/${account_id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora

https://api.cloudflare.com/client/v4/accounts/${account_id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora

℠℠I’d bet my bald head that he means the model keeps training on chat-data instead...

A

Ali_hat•6/20/24, 6:53 AM

Yes, exactly. Which is not what I'd expect. Of course it's just an impression but it's a bit weird for me when I call the endpoint the first time (a decent response) and a second/third time with the same parameters (now it's a much shorter and less informative response).

AAli_hat Yes, exactly. Which is not what I'd expect. Of course it's just an impression bu...

℠

℠•6/20/24, 7:24 AM

That’s the randomness at play

℠

℠•6/20/24, 7:25 AM

Maybe you want to fine tune it and train it to provide more consistent answers in the form that you need?

℠

℠•6/20/24, 7:26 AM

Asking GPT the exact same question would also yield me sometimes very different answers - don’t know if that’s changed, as I haven’t been working too much with the chat models

℠

℠•6/20/24, 7:29 AM

In SD you can provide a fixed seed, which means you can keep reloading the same prompt until the style details are some you like. Then you can lock that seed in and continue working on variants/changes but instead of entirely different outputs, it stays in the vicinity of the original image

℠

℠•6/20/24, 7:30 AM

So you can make alts with a similar looking character in the same landscape and style, etc

℠

℠•6/20/24, 7:30 AM

Maybe you can do something akin to that with yours

℠℠Maybe you want to fine tune it and train it to provide more consistent answers i...

A

Ali_hat•6/20/24, 7:31 AM

Yes, that's eventually what I'd like to do. And getting different answers is usually fine. I had hoped to squeeze out a bit more without finetuning but I'm maybe being too naive.

℠℠In SD you can provide a fixed seed, which means you can keep reloading the same ...

A

Ali_hat•6/20/24, 7:31 AM

That's a good hint, I haven't experimented with using seed yet.

℠

℠•6/20/24, 7:32 AM

As it sounds like you don’t really need a dynamic bot for long conversation, but just one that does single a Q&A at a time

P

Phonebox•6/21/24, 9:26 PM

Is anyone running langchain on workers? Curious to get some insights into examples of costs

Mmichelle hm sending you a dm

H

hydroxide•6/22/24, 3:29 AM

Sure, feel free to DM me

S

sarme•6/22/24, 4:26 PM

Is there any plan to support an image embedding model in the api?

V

vrballer•6/23/24, 12:56 AM

anyone know of an open source model better than whisper? If not anyone tried training it?

Vvrballer anyone know of an open source model better than whisper? If not anyone tried tra...

I

Isaac McFadyen•6/24/24, 2:34 AM

Depends. Nvidia's Parakeet is quite good at noisy audio (far better than Whisper in my experience). Haven't tried training/finetuning it but the base model is quite good.

I

Isaac McFadyen•6/24/24, 2:34 AM

Parakeet has two versions - one with no punctuation that is best for accuracy (but is missing punctuation) and one that has punctuation like Whisper that has a little less accuracy.

I

Isaac McFadyen•6/24/24, 2:35 AM

If you go with the one that doesn't have punctuation there are other models designed to take text as input and add punctuation that you can run on the text after.

G

German•6/24/24, 5:17 PM

What's the highest token count LLM available on workers ai?

Hello, do you plan to add Stable Diffusion 3 Medium to Model Catalog?

Similar Threads

Similar Threads

Similar Threads