Serverless docker image deployment
Hi,
I finetuned a lora from llama 3.2 3B using unsloth. and want to deploy that on serverless.
Using vLLM with merged model degrades the performance too much to be of use. I then, followed instructions from this link https://github.com/runpod-workers/worker-template/tree/main and created a serverless endpoint using the docker image. but it keeps on initializing and does not complete one job. job remains in queue.
I might be missing something. I also don't have much experience with docker. I might be making a mistake there. But I did test the docker locally before deploying. I would appreciate any help regarding this.
I finetuned a lora from llama 3.2 3B using unsloth. and want to deploy that on serverless.
Using vLLM with merged model degrades the performance too much to be of use. I then, followed instructions from this link https://github.com/runpod-workers/worker-template/tree/main and created a serverless endpoint using the docker image. but it keeps on initializing and does not complete one job. job remains in queue.
I might be missing something. I also don't have much experience with docker. I might be making a mistake there. But I did test the docker locally before deploying. I would appreciate any help regarding this.
GitHub
| A simple worker that can be used as a starting point to build your own custom RunPod Endpoint API worker. - runpod-workers/worker-template
| A simple worker that can be used as a starting point to build your own custom RunPod Endpoint API worker. - runpod-workers/worker-template