Rope scaling JSON not working
When I try to use rope scaling, with the JSON that works fine in my own vLLM... it errors out on serverless. I tried setting it to just 'type' also but this produces the same error.
{"factor":4,"original_max_position_embeddings":32768,"rope_type":"yarn"}
Here is the log:
2024-11-10T10:54:29.062414854Z engine.py :114 2024-11-10 10:54:29,062 Error initializing vLLM engine: string indices must be integers
....
2024-11-10T10:54:29.064682309Z rope_type = rope_scaling["type"]
2024-11-10T10:54:29.064683918Z TypeError: string indices must be integers
1 Reply
Unknown User•12mo ago
Message Not Public
Sign In & Join Server To View