Devcontainer Template Ignores GPU Limits (All GPUs Visible)

Hey folks

I'm running into an issue with GPU isolation in two different Kubernetes-based Coder templates:

Template A: Uses the Kubernetes (Deployment) template → GPU isolation works as expected. If I select 1 GPU, the container only sees 1 via nvidia-smi.

Template B: Uses the Kubernetes (Devcontainer) template → Even when I select 1 GPU, the container sees all available GPUs on the host.

Both templates configure GPU resources like this:

resources {
  requests = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
  limits = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
}

resources {
  requests = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
  limits = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
}

One key difference is that in the Devcontainer template, I had to add the following to the

security_context

security_context

security_context {
  run_as_user = 0
  privileged  = true
}

security_context {
  run_as_user = 0
  privileged  = true
}

I suspect this might be allowing the container to bypass Kubernetes’ GPU isolation, but I’m not sure how to safely lock it down and still allow the build process to succeed.

Has anyone dealt with this before? Is there a way to use envbuilder + GPU isolation without needing to run as root/privileged?

Any pointers would be much appreciated

Coder.com•12mo ago•

16 replies

Mikel

Devcontainer Template Ignores GPU Limits (All GPUs Visible)

Hey folks

resources {
  requests = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
  limits = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
}

resources {
  requests = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
  limits = {
    "nvidia.com/gpu" = data.coder_parameter.gpu.value
  }
}

One key difference is that in the Devcontainer template, I had to add the following to the

security_context

security_context

security_context {
  run_as_user = 0
  privileged  = true
}

security_context {
  run_as_user = 0
  privileged  = true
}

Devcontainer Template Ignores GPU Limits (All GPUs Visible)

Similar Threads

Devcontainer Template Ignores GPU Limits (All GPUs Visible)

Similar Threads

Similar Threads

Similar Threads