Coder.comC
Coder.com•10mo ago•
16 replies
Mikel

Devcontainer Template Ignores GPU Limits (All GPUs Visible)

Hey folks 👋

I'm running into an issue with GPU isolation in two different Kubernetes-based Coder templates:

Template A: Uses the Kubernetes (Deployment) template → GPU isolation works as expected. If I select 1 GPU, the container only sees 1 via nvidia-smi.

Template B: Uses the Kubernetes (Devcontainer) template → Even when I select 1 GPU, the container sees all available GPUs on the host.

Both templates configure GPU resources like this:
resources {
  requests = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
  limits = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
}

One key difference is that in the Devcontainer template, I had to add the following to the security_context:
security_context {
  run_as_user = 0
  privileged  = true
}

I suspect this might be allowing the container to bypass Kubernetes’ GPU isolation, but I’m not sure how to safely lock it down and still allow the build process to succeed.

Has anyone dealt with this before? Is there a way to use envbuilder + GPU isolation without needing to run as root/privileged?

Any pointers would be much appreciated 🙏
Was this page helpful?