Seems like seed, temperature etc doesn't have any effect when using `@hf/nousresearch/hermes-2-pro-m

J

James•7/20/24, 8:17 PM

The TypeScript story with Workers AI really needs a lot of improvements. It'd be good to see this fixed soon for a product that advertises GA. https://github.com/cloudflare/workerd/issues/2418

M

Mitya•7/20/24, 11:51 PM

Could someone tell me what the point of this return string is (adapted from this example)? It doesn't form the AI response, so what's it for?

const updateKvValue = async ({ key, value }: { key: string; value: string }) => {
    await env.KV.put(key, value);
    return `Successfully updated key-value pair in database`; //<-- this
};

const updateKvValue = async ({ key, value }: { key: string; value: string }) => {
    await env.KV.put(key, value);
    return `Successfully updated key-value pair in database`; //<-- this
};

const updateKvValue = async ({ key, value }: { key: string; value: string }) => {
    await env.KV.put(key, value);
    return `Successfully updated key-value pair in database`; //<-- this
};

const updateKvValue = async ({ key, value }: { key: string; value: string }) => {
    await env.KV.put(key, value);
    return `Successfully updated key-value pair in database`; //<-- this
};

MMitya Could someone tell me what the point of this return string is (adapted from [thi...

R

RaylightOP•7/21/24, 7:08 PM

Embedded function calling is (currently) designed as an all-in-one solution. runWithTools maintains and adds to a copy of the message history internally, and the final output is always a plain response. (I.e. not a tool call.) When the model's response is a tool call, the matching user-defined function is called and both the tool call and the function's (stringified) return value is appended to the history.

In your example, the internal message history might look something like this after the first "iteration".

  { role: "system", content: "Put user given values in KV" },
  { role: "user", content: "Set the value of banana to yellow." },
  { role: "assistant", content: '{"arguments":{"key":"banana","value":"yellow"},"name":"KV update"}' },
  { role: "tool", "content": '"Successfully updated key-value pair in database."', "name": "KV update" }

  { role: "system", content: "Put user given values in KV" },
  { role: "user", content: "Set the value of banana to yellow." },
  { role: "assistant", content: '{"arguments":{"key":"banana","value":"yellow"},"name":"KV update"}' },
  { role: "tool", "content": '"Successfully updated key-value pair in database."', "name": "KV update" }

  { role: "system", content: "Put user given values in KV" },
  { role: "user", content: "Set the value of banana to yellow." },
  { role: "assistant", content: '{"arguments":{"key":"banana","value":"yellow"},"name":"KV update"}' },
  { role: "tool", "content": '"Successfully updated key-value pair in database."', "name": "KV update" }

  { role: "system", content: "Put user given values in KV" },
  { role: "user", content: "Set the value of banana to yellow." },
  { role: "assistant", content: '{"arguments":{"key":"banana","value":"yellow"},"name":"KV update"}' },
  { role: "tool", "content": '"Successfully updated key-value pair in database."', "name": "KV update" }

Inference is run again with this as input and the result might be something like 'I have successfully updated the key-value pair for you. The value of "banana" is now set to "yellow" in the database.'

S

sam•7/22/24, 6:35 AM

Just to confirm, if I use a text generation beta model (@cf/meta/llama-3-8b-instruct) it will be completely free right?

E

Erisa•7/22/24, 8:52 AM

yes

0

0xCC•7/23/24, 1:04 AM

Hello, I have hit the daily usage limit on worker AI API call, when it would be the usage limit renew?
It is the second day - but i still see the usage is out of limit

JJames Is the `@cf/llava-hf/llava-1.5-7b-hf` model not working right now? Seems to be t...

K

Korcan•7/23/24, 4:51 PM

Hello, are there any updates about this? Experiencing the same issue with

@cf/unum/uform-gen2-qwen-500m

@cf/unum/uform-gen2-qwen-500m

@cf/unum/uform-gen2-qwen-500m

@cf/unum/uform-gen2-qwen-500m

Mmichelle that issue was fixed - what error are you seeing with uform?

K

Korcan•7/23/24, 6:09 PM

InferenceUpstreamError: undefined: undefined

InferenceUpstreamError: undefined: undefined

InferenceUpstreamError: undefined: undefined

InferenceUpstreamError: undefined: undefined

at Ai._parseError (cloudflare-internal:ai-api:74:20)
at async Ai.run (cloudflare-internal:ai-api:55:23)

at Ai._parseError (cloudflare-internal:ai-api:74:20)
at async Ai.run (cloudflare-internal:ai-api:55:23)

at Ai._parseError (cloudflare-internal:ai-api:74:20)
at async Ai.run (cloudflare-internal:ai-api:55:23)

at Ai._parseError (cloudflare-internal:ai-api:74:20)
at async Ai.run (cloudflare-internal:ai-api:55:23)

00xCC Hello, I have hit the daily usage limit on worker AI API call, when it would be ...

K

Kathy•7/24/24, 8:25 PM

All limits reset daily at 00:00 UTC https://developers.cloudflare.com/workers-ai/platform/pricing/

Cloudflare Docs

Pricing · Cloudflare Workers AI docs

Workers AI will begin billing for usage on non-beta models after April 1, 2024.

KKathy All limits reset daily at 00:00 UTC https://developers.cloudflare.com/workers-ai...

0

0xCC•7/24/24, 9:37 PM

Yes. It was test after sharing account Id with Micheal

A

akazwz•7/25/24, 3:43 AM

Hi, team, do u have plan to add more Text Embeddings models that support other languages, looks like the BAAI model have Multilingual languages support https://huggingface.co/BAAI/bge-base-en-v1.5

BAAI/bge-base-en-v1.5 · Hugging Face

RRaylight Embedded function calling is (currently) designed as an all-in-one solution. run...

M

Mitya•7/25/24, 3:49 PM

Thank you for this (and sorry for the delayed response). I'm still a little hazy, however, on how the call to my user-defined function influences (if at all) the AI response. In the CF docs there's an example of an embedded function to get the next N days' weather. It seems as though the AI response takes account of the API response, but it does not repeat it verbatim. Is there a way to have the user-defined function form the AI response, or am I still missing something? (To be clear, I want the user-defined function to control the response, rather than AI deriving its own conversational response.)

MMitya Thank you for this (and sorry for the delayed response). I'm still a little hazy...

R

RaylightOP•7/25/24, 5:55 PM

Right now your option is pretty much to use "traditional" function calling. When you call runWithTools it's the LLM that generates the response, i.e. you can't make it return the result of your user-defined function(s).

A

AngusMa•7/27/24, 5:14 AM

How to use the Model Search API with task query param? I tried

?task=c329a1f9-323d-4e91-b2aa-582dd4188d34

?task=c329a1f9-323d-4e91-b2aa-582dd4188d34

?task=c329a1f9-323d-4e91-b2aa-582dd4188d34

?task=c329a1f9-323d-4e91-b2aa-582dd4188d34 and

?task=Text%20Generation

?task=Text%20Generation

?task=Text%20Generation

?task=Text%20Generation, both of them returned error.

{
   "errors":[
      {
         "message":"Server Error",
         "code":1000
      }
   ],
   "success":false,
   "result":{},
   "messages":[]
}

{
   "errors":[
      {
         "message":"Server Error",
         "code":1000
      }
   ],
   "success":false,
   "result":{},
   "messages":[]
}

{
   "errors":[
      {
         "message":"Server Error",
         "code":1000
      }
   ],
   "success":false,
   "result":{},
   "messages":[]
}

{
   "errors":[
      {
         "message":"Server Error",
         "code":1000
      }
   ],
   "success":false,
   "result":{},
   "messages":[]
}

S

scotto•7/27/24, 12:52 PM

Any plans to support the new llama 3.1?

D

DaniFoldi•7/27/24, 1:25 PM

I have some good news for you!
https://blog.cloudflare.com/meta-llama-3-1-available-on-workers-ai

The Cloudflare Blog

Meta Llama 3.1 now available on Workers AI

Cloudflare is excited to be a launch partner with Meta to introduce Workers AI support for Llama 3.1.

S

scotto•7/27/24, 2:12 PM

Oh that made my day

S

scotto•7/27/24, 2:12 PM

Thanks

K

Kraii•7/27/24, 8:35 PM

is the rest api reference accurate, cause it says that llama 3.1 accepts a prompt of up to 131k characters but when calling the api with a prompt of over 30k ish characters it responds with an empty message

K

Kraii•7/27/24, 8:35 PM

K

Kraii•7/27/24, 8:36 PM

KKraii is the rest api reference accurate, cause it says that llama 3.1 accepts a promp...

R

RaylightOP•7/28/24, 8:39 AM

There's also a token limit that depends on which model you use. Looks like llama 3.1 on Cloudflare currently has a limit of ~6000 tokens.

K

Kraii•7/29/24, 7:19 AM

yea but i cant find any info on the token limits

KKraii yea but i cant find any info on the token limits

R

RaylightOP•7/29/24, 2:34 PM

Yeah I know.. With a few exceptions, you won't be able to find it. Ideally, one should be able to get info about limits, token counts, default prompt, supported params/features etc etc via methods and API calls. In any case, here's a list of tokens limits that covers most of the models, based on my own tests: https://gist.github.com/Raylight-PWL/e92b298850625c80efa6ca4ddd206662

K

Kraii•7/29/24, 7:37 PM

ow that's great

K

Kraii•7/29/24, 7:37 PM

thank you very much :firSweet:

R

RaylightOP•7/29/24, 10:01 PM

Found a somewhat weird bug: models that start with

@hf/thebloke

@hf/thebloke

@hf/thebloke

@hf/thebloke won't generate a response if max_tokens is set to 597 or higher. (Fails with "3025: Unknown internal error".)

D

David Liu•7/30/24, 5:04 PM

Hey Everyone!
We’re super excited to launch this awesome demo playground where you can use audio, video, images, and text to create multimodal AI applications. We ask that you take it for a spin, try it out and post what you build. There are some preloaded examples to get you started but feel free to build whatever you like! We’ll retweet your posts on X. Try it here https://multi-modal.ai.cloudflare.com/

D

David Liu•7/30/24, 5:05 PM

Here's me using it to turn an image from Cloudflare's website to HTML code

U

user•7/30/24, 6:50 PM

Uhm, did the

runWithTools

runWithTools

runWithTools

runWithTools syntax change?

U

user•7/30/24, 6:53 PM

When I call

runWithTools

runWithTools

runWithTools

runWithTools I get

Error in runWithTools: Error in runAndProcessToolCall: Bad input: must have required property 'prompt', must have required property 'name', must have required property 'description', must have required property 'parameters', must have required property 'type', must have required property 'properties', must match exactly one schema in oneOf, must match exactly one schema in oneOf,

Error in runWithTools: Error in runAndProcessToolCall: Bad input: must have required property 'prompt', must have required property 'name', must have required property 'description', must have required property 'parameters', must have required property 'type', must have required property 'properties', must match exactly one schema in oneOf, must match exactly one schema in oneOf,

Error in runWithTools: Error in runAndProcessToolCall: Bad input: must have required property 'prompt', must have required property 'name', must have required property 'description', must have required property 'parameters', must have required property 'type', must have required property 'properties', must match exactly one schema in oneOf, must match exactly one schema in oneOf,

Error in runWithTools: Error in runAndProcessToolCall: Bad input: must have required property 'prompt', must have required property 'name', must have required property 'description', must have required property 'parameters', must have required property 'type', must have required property 'properties', must match exactly one schema in oneOf, must match exactly one schema in oneOf,

U

user•7/30/24, 7:08 PM

i cant seem to find where the "Bad input" part comes from, is it an upstream issue rather than an ai-utils issue?

E

Egor•7/31/24, 10:42 AM

Hello everyone

! Please tell me if it is possible to understand how many Neurons were used in each request to the model?

Uuser When I call `runWithTools` I get `Error in runWithTools: Error in runAndProcessT...

U

user•7/31/24, 11:44 PM

Sorry for the ping, @Dhravya Shah, but you're the only know I can think of who can help with this xD

U

user•7/31/24, 11:45 PM

same code snippet as last time produces this error, no changes were made

U

user•7/31/24, 11:48 PM

Qwen

U

user•7/31/24, 11:48 PM

Still trying to figure out functions with qwen :ye:

U

user•7/31/24, 11:48 PM

@cf/qwen/qwen1.5-14b-chat-awq

@cf/qwen/qwen1.5-14b-chat-awq

@cf/qwen/qwen1.5-14b-chat-awq

@cf/qwen/qwen1.5-14b-chat-awq

DDhravya Shah looking into it, upstream issue, maybe I'm parsing it wrong or API is returning ...

U

user•7/31/24, 11:49 PM

My immediate guess is the upstream api changed and ai-utils is no longer sending the right data

U

user•7/31/24, 11:50 PM

because the only parts of the error that I could find in ai-utils was

Error in runWithTools: Error in runAndProcessToolCall:

Error in runWithTools: Error in runAndProcessToolCall:

Error in runWithTools: Error in runAndProcessToolCall:

Error in runWithTools: Error in runAndProcessToolCall: , making me think that maybe the "Bad input" part was a response from the api

U

user•7/31/24, 11:59 PM

is it maybe the tool parsing?

U

user•8/1/24, 12:00 AM

i dont see where the "prompt" property would come in as I don't see that documented anywhere, but everything else is properties of a tool

E

E S•8/1/24, 4:24 AM

https://tenor.com/view/dancing-cat-dance-cat-cat-meme-chinese-cat-gif-12629347036627000898

Tenor

K

konedi•8/2/24, 1:16 PM

Hi all, anyway to speed up workers ai through rest api? responses are taking up to 30s, while I have a local llama3.1 running on a rtx3080 and it only takes like 6s?

O

oakwhiz•8/2/24, 8:25 PM

What's the pricing for neurons on text to image generation?

Uuser i dont see where the "prompt" property would come in as I don't see that documen...

R

RaylightOP•8/2/24, 9:05 PM

These error messages can be quite cryptic. Omitting the property "type" inside a single parameter of a single tool would produce the same error, for example. In your example with

parameters: {}

parameters: {}

parameters: {}

parameters: {} I got past the error by changing it to

parameters: { type: "object", properties: {} }

parameters: { type: "object", properties: {} }

parameters: { type: "object", properties: {} }

parameters: { type: "object", properties: {} }.

RRaylight These error messages can be quite cryptic. Omitting the property "type" inside a...

U

user•8/2/24, 9:07 PM

Interesting.

Seems like seed, temperature etc doesn't have any effect when using `@hf/nousresearch/hermes-2-pro-m

Similar Threads

Similar Threads

Similar Threads