Chat completion (template) not working with VLLM 0.6.3 + Serverless

I deployed https://huggingface.co/xingyaoww/Qwen2.5-Coder-32B-Instruct-AWQ-128k model through the Serverless UI, setting max model context window to 129024 and quantization to awq. I deploy it using the lastest version of vllm (0.6.3) provided by runpod.

I ran into the following errors

Client-side

ChatCompletion(id=None, choices=None, created=None, model=None, object='error', service_tier=None, system_fingerprint=None, usage=None, code=400, message="expected token 'end of print statement', got 'name'", param=None, type='BadRequestError')

ChatCompletion(id=None, choices=None, created=None, model=None, object='error', service_tier=None, system_fingerprint=None, usage=None, code=400, message="expected token 'end of print statement', got 'name'", param=None, type='BadRequestError')

xingyaoww/Qwen2.5-Coder-32B-Instruct-AWQ-128k · Hugging Face

Runpod•16mo ago•

7 replies

xxxyyy

Chat completion (template) not working with VLLM 0.6.3 + Serverless

ChatCompletion(id=None, choices=None, created=None, model=None, object='error', service_tier=None, system_fingerprint=None, usage=None, code=400, message="expected token 'end of print statement', got 'name'", param=None, type='BadRequestError')

ChatCompletion(id=None, choices=None, created=None, model=None, object='error', service_tier=None, system_fingerprint=None, usage=None, code=400, message="expected token 'end of print statement', got 'name'", param=None, type='BadRequestError')

xingyaoww/Qwen2.5-Coder-32B-Instruct-AWQ-128k · Hugging Face

Chat completion (template) not working with VLLM 0.6.3 + Serverless

Similar Threads

Chat completion (template) not working with VLLM 0.6.3 + Serverless

Similar Threads

Similar Threads

Similar Threads