Runpod•7mo ago

Can't deploy Qwen/Qwen2.5-14B-Instruct-1M on serverless

Steps to reproduce:

Use Serverless vLLM quick deploy for Qwen/Qwen2.5-14B-Instruct-1M (image attached)
Proceed with default config.
Try and send a request.

Error:

2025-06-18T12:58:36.147823280Z INFO 06-18 12:58:36 [model_runner.py:1170] Starting to load model Qwen/Qwen2.5-14B-Instruct-1M...
2025-06-18T12:58:36.449947523Z engine.py:116  2025-06-18 12:58:36,449 Error initializing vLLM engine: FlashAttentionImpl.__init__() got an unexpected keyword argument 'layer_idx'

2025-06-18T12:58:36.147823280Z INFO 06-18 12:58:36 [model_runner.py:1170] Starting to load model Qwen/Qwen2.5-14B-Instruct-1M...
2025-06-18T12:58:36.449947523Z engine.py:116  2025-06-18 12:58:36,449 Error initializing vLLM engine: FlashAttentionImpl.__init__() got an unexpected keyword argument 'layer_idx'

2025-06-18T12:58:36.147823280Z INFO 06-18 12:58:36 [model_runner.py:1170] Starting to load model Qwen/Qwen2.5-14B-Instruct-1M...
2025-06-18T12:58:36.449947523Z engine.py:116  2025-06-18 12:58:36,449 Error initializing vLLM engine: FlashAttentionImpl.__init__() got an unexpected keyword argument 'layer_idx'

2025-06-18T12:58:36.147823280Z INFO 06-18 12:58:36 [model_runner.py:1170] Starting to load model Qwen/Qwen2.5-14B-Instruct-1M...
2025-06-18T12:58:36.449947523Z engine.py:116  2025-06-18 12:58:36,449 Error initializing vLLM engine: FlashAttentionImpl.__init__() got an unexpected keyword argument 'layer_idx'

How do I fix this?

I've been trying to troubleshoot this all morning. All help appreciated

EEllroy Steps to reproduce: 1. Use Serverless vLLM quick deploy for Qwen/Qwen2.5-14B-In...

Jason•6/18/25, 2:58 PM

Oh you need to set DUAL_CHUNK_FLASH_ATTNDUAL_CHUNK_FLASH_ATTN= DUAL_CHUNK_FLASH_ATTNDUAL_CHUNK_FLASH_ATTN
and VLLM_ALLOW_LONG_MAX_MODEL_LENVLLM_ALLOW_LONG_MAX_MODEL_LEN=

Jason•6/18/25, 2:59 PM

probably its not on runpod vllm yet