The total token limit at 131

I use vLLM and set max model length to 8000 a2048 but out is just 131 (total out + in ), although i have set max tokens to 2048. I try with 2 models and result is the same.

Runpod Join

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

21,906Members

Resources

Similar Threads

Was this page helpful?

Twitter GitHub Discord

Communities Docs About Terms Privacy

Star

Setup for Free

The total token limit at 131 - Runpod

Runpod•11mo ago•

20 replies

tanawatl

The total token limit at 131

I use vLLM and set max model length to 8000 a2048 but out is just 131 (total out + in ), although i have set max tokens to 2048. I try with 2 models and result is the same.