Cannot run Cmdr+ on serverless, CohereForCausalLM not supported

I'm getting this error for all Cmdr+ models on serverless:

Error initializing vLLM engine: Model architectures ['CohereForCausalLM'] are not supported for now.

Error initializing vLLM engine: Model architectures ['CohereForCausalLM'] are not supported for now.

Although in vLLM issues we see that CohereForCausalLM is supported

Runpod•2y ago•

7 replies

Cannot run Cmdr+ on serverless, CohereForCausalLM not supported

I'm getting this error for all Cmdr+ models on serverless:

Error initializing vLLM engine: Model architectures ['CohereForCausalLM'] are not supported for now.

Error initializing vLLM engine: Model architectures ['CohereForCausalLM'] are not supported for now.

Although in vLLM issues we see that CohereForCausalLM is supported

Join the Discord to ask follow-up questions and connect with the community

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

21,906 Members