Youre likely using a beta model

Uuser Youre likely using a beta model

D

DoctorU•9/15/24, 3:49 AM

Yes, I am using multiple beta models. Are they not counted?

U

userOP•9/15/24, 3:50 AM

Nope

U

userOP•9/15/24, 3:50 AM

For as long as theyre in beta you have inf usage

D

DoctorU•9/15/24, 3:50 AM

That's really cool! Thanks for clarifying this for me.

S

Stéphane Magnenat•9/16/24, 8:23 AM

Hello. Does anyone knows the neuron cost of the model

llama-3.1-8b-instruct-awq

llama-3.1-8b-instruct-awq

llama-3.1-8b-instruct-awq

llama-3.1-8b-instruct-awq? I could not find this information. I am particularly interested in knowing whether that one is significantly cheaper than

llama-3.1-8b-instruct-fp8

llama-3.1-8b-instruct-fp8

llama-3.1-8b-instruct-fp8

llama-3.1-8b-instruct-fp8. It seems that overall the neuron cost table is not up to date with the model list.

S

Stéphane Magnenat•9/16/24, 8:33 AM

Oh I see, sorry for the confusion!

S

Stéphane Magnenat•9/16/24, 8:34 AM

Do you know whether there will be a cost difference between the

awq

awq

awq

awq and the

fp8

fp8

fp8

fp8 once they are out of beta? Just to know whether it is worth trying to stick with the smaller weight one during development.

F

FelikZ•9/16/24, 10:49 AM

Hello, do you plan to add "deepseek-coder-v2" anytime soon? Thanks

Ccodebam 4.20240620.0

A

acusti•9/17/24, 5:29 PM

@kian i get this same type error with

4.20240909.0

4.20240909.0

4.20240909.0

4.20240909.0, the current most recent version of workers-types. however, i only started getting the error when i migrated from using the now deprecated

@cloudflare/ai

@cloudflare/ai

@cloudflare/ai

@cloudflare/ai package to using

await env.AI.run(…)

await env.AI.run(…)

await env.AI.run(…)

await env.AI.run(…) directly, so it seems like the

@cloudflare/ai

@cloudflare/ai

@cloudflare/ai

@cloudflare/ai types allow me to use one of the

BaseAiTextGenerationModels

BaseAiTextGenerationModels

BaseAiTextGenerationModels

BaseAiTextGenerationModels, whereas the types for the direct

AI

AI

AI

AI binding (which use function overloading for usage with all possible types of AI generation) are, in my case, forcing the

model

model

model

model argument to be type

BaseAiImageToTextModels

BaseAiImageToTextModels

BaseAiImageToTextModels

BaseAiImageToTextModels . i am using typescript v5.6.2.

edit: discord makes it hard to follow the context. the type error i am getting is:

Argument of type '"@cf/meta/llama-3-8b-instruct" | "@hf/thebloke/zephyr-7b-beta-awq" | "@hf/nousresearch/hermes-2-pro-mistral-7b" | "@hf/nexusflow/starling-lm-7b-beta" | "@cf/defog/sqlcoder-7b-2" | "@cf/openchat/openchat-3.5-0106"' is not assignable to parameter of type 'BaseAiImageToTextModels'.

Argument of type '"@cf/meta/llama-3-8b-instruct" | "@hf/thebloke/zephyr-7b-beta-awq" | "@hf/nousresearch/hermes-2-pro-mistral-7b" | "@hf/nexusflow/starling-lm-7b-beta" | "@cf/defog/sqlcoder-7b-2" | "@cf/openchat/openchat-3.5-0106"' is not assignable to parameter of type 'BaseAiImageToTextModels'.

Argument of type '"@cf/meta/llama-3-8b-instruct" | "@hf/thebloke/zephyr-7b-beta-awq" | "@hf/nousresearch/hermes-2-pro-mistral-7b" | "@hf/nexusflow/starling-lm-7b-beta" | "@cf/defog/sqlcoder-7b-2" | "@cf/openchat/openchat-3.5-0106"' is not assignable to parameter of type 'BaseAiImageToTextModels'.

Argument of type '"@cf/meta/llama-3-8b-instruct" | "@hf/thebloke/zephyr-7b-beta-awq" | "@hf/nousresearch/hermes-2-pro-mistral-7b" | "@hf/nexusflow/starling-lm-7b-beta" | "@cf/defog/sqlcoder-7b-2" | "@cf/openchat/openchat-3.5-0106"' is not assignable to parameter of type 'BaseAiImageToTextModels'.

(my code has an enum of models that can be used with retry logic in case one of the models doesn't give a good result)

H

hari•9/19/24, 5:59 AM

what are the ratelimits of workers ai beta models? can you help to understand these limits please?

V

vin•9/19/24, 2:53 PM

here is the document https://developers.cloudflare.com/workers-ai/platform/limits/

Cloudflare Docs

Limits | Cloudflare Workers AI docs

Workers AI is now Generally Available. We’ve updated our rate limits to reflect this.

S

scotto•9/25/24, 7:37 PM

plans / date to support llama 3.2 3B ?

S

scotto•9/25/24, 7:37 PM

ideal model for cloudflore (size)

A

Ashish Rawat•9/26/24, 6:21 AM

check here: https://github.com/cloudflare/cloudflare-docs/pull/17119/files

GitHub

Workers AI Birthday Week Updates by mchenco · Pull Request #17119 ·...

Summary
new pricing page and changelog
dash refresh in separate PR
l3.2 in separate PR

Screenshots (optional)

Documentation checklist

[x ] The documentation style guide has been adhered to.

C

cookie•9/26/24, 6:42 AM

already here https://playground.ai.cloudflare.com/

C

crazyjack12•9/26/24, 7:45 AM

https://forms.gle/h7FcaTF4Zo5dzNb68 not open yet

Google Forms: Sign-in

Access Google Forms with a personal Google account or Google Workspace account (for business use).

A

akazwz•9/26/24, 4:22 PM

K

Keebs•9/26/24, 5:21 PM

const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

anyone know why the local example code takes 25 seconds to respond?

K

Keebs•9/26/24, 5:34 PM

oh it might be the model

K

Keebs•9/26/24, 5:45 PM

no still slow 20 seconds minimum

K

Keebs•9/26/24, 5:45 PM

Response time: 716 milliseconds
[wrangler:inf] GET / 200 OK (739ms)

Response time: 716 milliseconds
[wrangler:inf] GET / 200 OK (739ms)

Response time: 716 milliseconds
[wrangler:inf] GET / 200 OK (739ms)

Response time: 716 milliseconds
[wrangler:inf] GET / 200 OK (739ms)

this is when i make a request to groq...

B

Bibi•9/26/24, 6:35 PM

Groq doesn’t use gpus so it’s obviously way faster but 25 seconds is indeed quite long

K

Keebs•9/26/24, 7:08 PM

well which other api isnt really the issue - cloudflare ai is :D

K

Keebs•9/26/24, 7:08 PM

just saying, a remote api request takes 500ms vs cloudflare ai, no matter which model, takes 20+ seconds

K

Keebs•9/26/24, 7:08 PM

and im not sure if that is me or if that is normal

B

Beyondo•9/26/24, 7:10 PM

Is it me or did the LLM pricing increase with the newer pricing model? Some models were $0.11 like mixtral, now $0.15 with that new bracket. Anyways, I don't think I'm using Workers AI anytime soon—But I did like a lot of the other updates like no longer being billed for service binding invocations (as it always should've been), so now I'm encouraged to use them rather than package/link every microservice as a library in a single build to avoid the previous anti-microarchitecture model lol.

KKeebs just saying, a remote api request takes 500ms vs cloudflare ai, no matter which ...

B

Beyondo•9/26/24, 7:13 PM

I've looked up

llama-3.1-8b-instruct

llama-3.1-8b-instruct

llama-3.1-8b-instruct

llama-3.1-8b-instruct, and they seem all beta models. Maybe that's why? Try another model

K

Keebs•9/26/24, 7:25 PM

alrady said that, doesnt matter which model

K

Keebs•9/26/24, 7:28 PM

(theyre basically all beta)

K

Keebs•9/26/24, 7:28 PM

      const response = await env.AI.run("@hf/meta-llama/meta-llama-3-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

      const response = await env.AI.run("@hf/meta-llama/meta-llama-3-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

      const response = await env.AI.run("@hf/meta-llama/meta-llama-3-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

      const response = await env.AI.run("@hf/meta-llama/meta-llama-3-8b-instruct", {
        prompt: "What is the origin of the phrase Hello, World",
      });

K

Keebs•9/26/24, 7:28 PM

K

Keebs•9/26/24, 7:30 PM

K

Keebs•9/26/24, 7:30 PM

vs groq (or whichever) api call

B

Bibi•9/26/24, 7:35 PM

What about calling it twice? Maybe it’s just the first run taking so long

BBibi What about calling it twice? Maybe it’s just the first run taking so long

K

Keebs•9/26/24, 7:41 PM

I called it about 30 times by now

KKeebs I called it about 30 times by now

B

Bibi•9/26/24, 7:46 PM

I mean calling several times in between reloads

K

Keebs•9/26/24, 7:46 PM

reloads?

K

Keebs•9/26/24, 7:46 PM

im just hitting curl 5 times

K

Keebs•9/26/24, 7:46 PM

it has nothing to do with how many times you do it

K

Keebs•9/26/24, 7:46 PM

and it should be fast the first time

KKeebs Click to see attachment

B

Bibi•9/26/24, 7:47 PM

It says reloading server here

BBibi It says reloading server here

K

Keebs•9/26/24, 7:53 PM

yes when you change code

K

Keebs•9/26/24, 7:53 PM

K

Keebs•9/26/24, 7:53 PM

thats openai

KKeebs yes when you change code

B

Bibi•9/26/24, 7:55 PM

Ye so you’re running the code multiple times per code change

B

Bibi•9/26/24, 7:55 PM

Because it says response time only once that’s why I’m wondering

K

Keebs•9/26/24, 7:55 PM

cause its one request

Similar Threads

Similar Threads

Similar Threads