What is the recommended System Req for Building Worker Base Image
I was trying to build a custom runpod/worker-vllm:base-0.3.1-cuda${WORKER_CUDA_VERSION} image, but my 16vCPU, 64GB RAM server crashed. What is the recommended system spec for this purpose
13 Replies
@Alpay Ariyak can probably advise on this and maybe add it to the docs.
Are you trying to build the worker image or the vllm base image
vllm base image
can i build using runpod cloud, i can understand these are containers, but is there any docker in docker kind of setup
You can only use docker on CPU pods not on GPU pods
since CPU pods are not that powerful, they are useless in this context
You can use CPU pods to build images, because people request the ability to do so but yes they obviously don't have a GPU, you can't use docker on GPU pods for security reasons.
Let me try, thanks for the info
I have added a template for docker.io/docker but i am not able to start a web terminal session
Check the logs for your workers by clicking on one of them, you can only access the web terminal while a worker is actually running
I was able to build the image(CUDA 11.8) using a GCE VM 24vCPU, 128 GB. I am not sure if this is required, i was using the max cap in gcp free tier
Also docker.io/docker image was not working because bash was not installed, so the ssh connection was failing
If someone is trying to use runpod CPU/GPU for this purpose use some ubuntu dind image or github actions runner image by the community
for CUDA 12.1, this configuration is not enough
Which configuration?
Also docker.io/docker image was not working because bash was not installed, so the ssh connection was failingSounds like you're using a bad base image, bash is always usually installed.
24vCPU, 128 GB is not enough for CUDA 12.1
The vCPU and system memory have nothing to do with the CUDA version, the system requirements depend on your application, not the CUDA version.
Add
--build-arg max_jobs=2 --build-arg nvcc_threads=8