Can you now run gemma 3 in the vllm container?
In the serverless, its seems im getting an error, any help on this
Jcan you send the error
YI deleted it but it seems bc gemma3 is a new model so transformer is relatively outdated afaik?
Jhmm did you use vllm? or only transformer?
Jyeah maybe its a good thing to check first for compatibility
YI used the preset vllm, llama 3.2b worked but the new gemma 3 didnt
DvLLM needs to publish an update first unfortunately
DYou can use vLLM directly from the main branch, but that's not super easy if you're using our vLLM template iirc
JI think we can update and build our own vllm template from that vllm-worker repo in github easily
JJust update the requirement.txt or wherever installs the vllm
BLooks like vllm v0.8.0 added gemma3 support will the serverless vllm be updated soon?
Jusually its delayed, so probably a few days/ weeks late
AHi, i have the same issue , have you resolved it? , if then please help me out with it too
YI used ollana
YOllama
AOkay
Jyes i think vllm is updated
Jalready
JI deployed an end point to try to call gemma3:4b but nothing is happening when I call it, anybody managed?
JYes it works
Jjust now i've tried for you!
Output-from-gemma3.txt5.42KB
Jyou need access from hf + hf token to access it
Juse vllm to configure + check the allow remote code options in the config ( in runpod menu when configuring vllm )
Jsecond image is the next page
