hi all, how do i track model chat token usage and other numbers so that i know how much i should bil
hi all, how do i track model chat token usage and other numbers so that i know how much i should bill users ? is there an built-in rest api
JTracking usage of things is a common use-case for Analytics Engine: https://developers.cloudflare.com/analytics/analytics-engine/
Cloudflare Docs
Workers Analytics Engine provides unlimited-cardinality analytics at scale, via a built-in API to write data points from Workers, and a SQL API to query that data.
DCloudflare Docs
Cloudflare's AI Gateway allows you to gain visibility and control over your AI apps. By connecting your apps to AI Gateway, you can gather insights on how people are using your application with analytics and logging and then control how your application scales with features such as caching, rate limiting, as well as request retries, model fallba...
Uhello, i'm getting this error:
binding ai of type ai failed to generate: internal error; please try again later or contact support [code: 10021]✘ [ERROR] A request to the Cloudflare API (/accounts/xxx/workers/scripts/xxx/versions) failed.
U╰─± wrangler -v
wrangler 3.99.0
-------------------
i didn't change anything in the code, and it was working fine a few hours ago
wrangler 3.99.0-------------------
i didn't change anything in the code, and it was working fine a few hours ago
JCan you share what your AI binding looks like in your
wrangler.tomlnevermind, I can reproduce
Jgoing to escalate
UThanks James!
Sjust wanted to tag on that im also experiencing this @James and thankful im not the only one haha
JThe team are aware and actively looking into this, thanks folks
UYes it does tysm!
VWorkers AI's OpenAI compatible endpoint is missing the
(screenshot is returned)
index(screenshot is returned)
SHey guys, I am trying to use this text generation model but I am getting a typescript error:
const response = await context.env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", { messages });
No overload matches this call.
The last overload gave the following error.
Argument of type '"@cf/meta/llama-3.3-70b-instruct-fp8-fast"' is not assignable to parameter of type 'BaseAiImageToTextModels'.ts(2769)
index.d.ts(3720, 3): The last overload is declared here.
Has anyone been through this before? I was able to find an open issue on github but the PR is still open so I assume they haven't pushed a fix for it
const response = await context.env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", { messages });
No overload matches this call.
The last overload gave the following error.
Argument of type '"@cf/meta/llama-3.3-70b-instruct-fp8-fast"' is not assignable to parameter of type 'BaseAiImageToTextModels'.ts(2769)
index.d.ts(3720, 3): The last overload is declared here.
Has anyone been through this before? I was able to find an open issue on github but the PR is still open so I assume they haven't pushed a fix for it
JThat should be fixed with the most recent workers-types version, via https://github.com/cloudflare/workerd/pull/3242. Are you up to date?
BHey, the docs are sorta confusing, Are we able to pay for more neurons?
If so, is it monthly, could you provide some help about the neurons?
If so, is it monthly, could you provide some help about the neurons?
BSince the docs say we have 100k requests, but ai says we have 10k neurons
JWorkers AI pricing is all kinds of confusing right now. Docs talk about tokens which is the “new” method that was set to go into action on November 1st but never happened, dashboard talks about neurons, old marketing talks about neurons, etc.
JHopefully we can get some more official clarification and updates soon
BHow are we able to get more neurons?
JNeurons aren’t a thing anymore - it’s token based now, but nothing in the UI reflects that
JIt’s usage based though - you just pay for how many you use
BBut what about the "5k/10k daily neurons used"?
JThat’s outdated, hence the confusion
JUpgrade to workers paid and then it’s usage based
BYeah I'm pretty confused
BAh, by the way by every token is it per character or per word?
BLike what do you guys define a token as
JI couldn’t tell you, frankly. Docs are confusing and there’s no usage reports in the dash for tokens to clarify
B0.000015$ per token, does that mean per word?
JHopefully we’ll get some clarification soon - this was meant to happen November 1st but didn’t
JI don’t know
BOh alright
BI just need to know before we start paying
JSorry I can’t be of more help. We’re (the community) all just as confused as you
BAs we don't wanna go around spending too much, though 1 token per word seems more reasonable
BYeah it's alright thank you alot
CThis is a more generic AI/LLM question, tokens are their sort of base building blocks, not always matching up with a single word: https://www.johno.com/tokens-vs-words
BThank you!
But for example, if I used only latin and symbols, how much would a token be?
But for example, if I used only latin and symbols, how much would a token be?
CDepends on the model you use/it's tokenizer
CFor example, OpenAI's tokenizers are like this: https://platform.openai.com/tokenizer, cool visulization to play around with
Cone for a model that is on Worker's AI, Lllama 3's https://lunary.ai/llama3-tokenizer
Mistral and Gemma's linked on that one too
Mistral and Gemma's linked on that one too
BWhat about mixtral's?
IMixtral uses the Mistral tokenizer.
ISo it'll be the same as the Mistral button on that page.
BOh alright thanks
KI noticed that llama 3.1-8B is out of beta as of the past month or so, but I don't see any reference to this in the Workers AI changelog. imo a billing-impacting change like this needs to be mentioned in the changelog
Cloudflare Docs
Review recent changes to Cloudflare Workers AI.
BI don't see any reranking models in the models offered by CF. Is there any plan to add one or more in the near future? BAAI embedding models are offered, it would be great to also offer their reranking model: https://huggingface.co/BAAI/bge-reranker-base.
Znot working for me, what it should be? I tried uint8, base64 but nothing seems to work
LHi, do you know how to get more than only 5 labels and scores as the result?
