```js const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", { prompt: "What is
anyone know why the local example code takes 25 seconds to respond?
Koh it might be the model
Kno still slow 20 seconds minimum
Kthis is when i make a request to groq...
BGroq doesn’t use gpus so it’s obviously way faster but 25 seconds is indeed quite long
Kwell which other api isnt really the issue - cloudflare ai is :D
Kjust saying, a remote api request takes 500ms vs cloudflare ai, no matter which model, takes 20+ seconds
Kand im not sure if that is me or if that is normal
BIs it me or did the LLM pricing increase with the newer pricing model? Some models were $0.11 like mixtral, now $0.15 with that new bracket. Anyways, I don't think I'm using Workers AI anytime soon—But I did like a lot of the other updates like no longer being billed for service binding invocations (as it always should've been), so now I'm encouraged to use them rather than package/link every microservice as a library in a single build to avoid the previous anti-microarchitecture model lol.
BI've looked up
llama-3.1-8b-instruct, and they seem all beta models. Maybe that's why? Try another model
Kalrady said that, doesnt matter which model
K(theyre basically all beta)
KK
K
Kvs groq (or whichever) api call
BWhat about calling it twice? Maybe it’s just the first run taking so long
KI called it about 30 times by now
BI mean calling several times in between reloads
Kreloads?
Kim just hitting curl 5 times
Kit has nothing to do with how many times you do it
Kand it should be fast the first time
BIt says reloading server here
Kyes when you change code
K
Kthats openai
BYe so you’re running the code multiple times per code change
BBecause it says response time only once that’s why I’m wondering
Kcause its one request
K
Kim not sure how to tell you this, but its not that lol
BAh I see now
BWell I thought that maybe it has to load the model into memory the first time it runs on a worker so it takes so long and following requests are faster
BBut ye that’s not the case
Kits not loading anything into any memory
BHow is it supposed to run the model then
BAt some point that has to be done
BEven if it’s not right before you access it
Kyou think they load an instance of a llm into some memory somewhere for every worker that invokes it?
BOnly if it wasn’t in memory already, ya
B“Some” memory being the ram of the machine your worker code runs on
BOr rather its vram
Kand then where is it executed?
Kon a vcpu in the worker?
BWhy a cpu? Cf has gpus in their machines
BHuh
BWhy thumbs up
Ddoes flux work for anyone?
i'm getting
i'm getting
AiExternal: Couldn't fetch external AI provider response (500, Internal Server Error)Dusing the code provided in the docs here: https://developers.cloudflare.com/workers-ai/models/flux-1-schnell
