Jehex Comments - Answer Overflow

Ok 🙂 about my issue with DeepSeek distilled r1, seems the prompt system is weird and tricky to use, if anyone know a good uncensored model to use vllm let me know ( I’m using llama 3.3 but it’s too censored )

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Yep exact but you can also pre configure the pre built docker image from the env variables right ?

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Is they’re any difference between using the fast deployment > vllm or using pre built the docker image

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Yep, I basically create a template from https://github.com/runpod-workers/worker-vllm then modify models etc. from env variables right and also modify the few lines of code for be able to call the openai api

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Nice thanks you for theses infos

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

hm im not very familiar with the openai sdk, is it something to configure during the creation of the serverless endpoint ( with vllm ) ?

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Ah ok, do you have an example of correct input for this model ?

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

{ "input": { "messages": [ { "role": "system", "content": "Your are an ai assistant." }, { "role": "user", "content": "Explain llm models" } ], "max_tokens": 3000, "temperature": 0.7, "top_p": 0.95, "n": 1, "stream": false, "stop": [], "presence_penalty": 0, "frequency_penalty": 0, "logit_bias": {}, "user": "utilisateur_123", "best_of": 1, "echo": false } }

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

my input look like this now :

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

basically used this model casperhansen/deepseek-r1-distill-qwen-32b-awq with vllm and runpod serverless, except lower the model max lenght to 11000 I didnt modify any others settings

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Yep increase to 3000 but still getting a short " thinking " answer 😦

108 replies

RRunPod

•Created by Lattus on 1/22/2025 in #⚡｜serverless

Serverless deepseek-ai/DeepSeek-R1 setup?

Thanks ! Will let you know if it’s work

108 replies

RRunPod