just saying, a remote api request takes 500ms vs cloudflare ai, no matter which model, takes 20+ sec
just saying, a remote api request takes 500ms vs cloudflare ai, no matter which model, takes 20+ seconds
Kand im not sure if that is me or if that is normal
BIs it me or did the LLM pricing increase with the newer pricing model? Some models were $0.11 like mixtral, now $0.15 with that new bracket. Anyways, I don't think I'm using Workers AI anytime soon—But I did like a lot of the other updates like no longer being billed for service binding invocations (as it always should've been), so now I'm encouraged to use them rather than package/link every microservice as a library in a single build to avoid the previous anti-microarchitecture model lol.
BI've looked up
llama-3.1-8b-instruct, and they seem all beta models. Maybe that's why? Try another model
Kalrady said that, doesnt matter which model
K(theyre basically all beta)
KK
K
Kvs groq (or whichever) api call
BWhat about calling it twice? Maybe it’s just the first run taking so long
KI called it about 30 times by now
BI mean calling several times in between reloads
Kreloads?
Kim just hitting curl 5 times
Kit has nothing to do with how many times you do it
Kand it should be fast the first time
BIt says reloading server here
Kyes when you change code
K
Kthats openai
BYe so you’re running the code multiple times per code change
BBecause it says response time only once that’s why I’m wondering
Kcause its one request
K
Kim not sure how to tell you this, but its not that lol
BAh I see now
BWell I thought that maybe it has to load the model into memory the first time it runs on a worker so it takes so long and following requests are faster
BBut ye that’s not the case
Kits not loading anything into any memory
BHow is it supposed to run the model then
BAt some point that has to be done
BEven if it’s not right before you access it
Kyou think they load an instance of a llm into some memory somewhere for every worker that invokes it?
BOnly if it wasn’t in memory already, ya
B“Some” memory being the ram of the machine your worker code runs on
BOr rather its vram
Kand then where is it executed?
Kon a vcpu in the worker?
BWhy a cpu? Cf has gpus in their machines
BHuh
BWhy thumbs up
Ddoes flux work for anyone?
i'm getting
i'm getting
AiExternal: Couldn't fetch external AI provider response (500, Internal Server Error)Dusing the code provided in the docs here: https://developers.cloudflare.com/workers-ai/models/flux-1-schnell
RIt takes a long time because you're not using streaming. You're basically waiting for the LLM to generate the whole text before returning a response.
Kit takes 20 seconds to write a sentence?
Khow’s any other external api then 30x faster?
Kthat still doesn’t make any sense
RTry the playground (https://playground.ai.cloudflare.com/) with your prompt and you'll get a feel for the perfomance.
Kyeah takes a second
