K
Kthats openai
BYe so you’re running the code multiple times per code change
BBecause it says response time only once that’s why I’m wondering
Kcause its one request
K
Kim not sure how to tell you this, but its not that lol
BAh I see now
BWell I thought that maybe it has to load the model into memory the first time it runs on a worker so it takes so long and following requests are faster
BBut ye that’s not the case
Kits not loading anything into any memory
BHow is it supposed to run the model then
BAt some point that has to be done
BEven if it’s not right before you access it
Kyou think they load an instance of a llm into some memory somewhere for every worker that invokes it?
BOnly if it wasn’t in memory already, ya
B“Some” memory being the ram of the machine your worker code runs on
BOr rather its vram
Kand then where is it executed?
Kon a vcpu in the worker?
BWhy a cpu? Cf has gpus in their machines
BHuh
BWhy thumbs up
Ddoes flux work for anyone?
i'm getting
i'm getting
AiExternal: Couldn't fetch external AI provider response (500, Internal Server Error)using the code provided in the docs here: https://developers.cloudflare.com/workers-ai/models/flux-1-schnell
RIt takes a long time because you're not using streaming. You're basically waiting for the LLM to generate the whole text before returning a response.
Kit takes 20 seconds to write a sentence?
Khow’s any other external api then 30x faster?
Kthat still doesn’t make any sense
RTry the playground (https://playground.ai.cloudflare.com/) with your prompt and you'll get a feel for the perfomance.
Kyeah takes a second
Kpick meta llama
prompt: What is the origin of the phrase Hello, World
prompt: What is the origin of the phrase Hello, World
Kalright you might be right
Kit’s just super ass slow lol
Kbut streaming it won’t make the response faster, just streamed
FI'm using Llama via api.cloudflare with Bearer Auth. Where can I check my usage? If I exceed 10,000 free neurons, am I charged automatically or does it stop working?
BIt stops working
BAnd their pricing is no longer in neurons
BCloudflare Docs
Workers AI is included in both the Free and Paid Workers plans and is priced based on model task, model size, and units.
Rneurons may have been the worst idea ever conceptualized
Kbetter :)
Koh btw. in the worker binding
Kasync env.AI.run()
Kis there docs on this, i cant find them...
Kim curious if theres a seed param like openai offers in the latest models
Kto get more deterministic results
Kalmost looks like sending a seed param worked somehow
RCheck the model page(s) (https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct-fast#Parameters). Beware that for the most part, models within a category list the same set of parameters. E.g. @hf/nousresearch/hermes-2-pro-mistral-7b doesn't support temperature or seed, even though those parameters are listed on the model page. (Reported it here workers-ai a while ago. Never got any response so no idea if it's intended to be that way). Also, expect some quirks. E.g. a subset of the models will break if you set max_tokens to 597 or higher (Reported here workers-ai)