AMD pods don't properly support GPU memory allocation
Hello! I've been trying to build a ROCm/HIP-based package to run on RunPod's ROCm-templated pods (or in a custom-built container/template), and I ran into memory issues that I believe I've tracked down to how RunPod is starting up docker containers.
In particular, pinned memory allocation fails with a misleading
I was able to do some testing on compute infrastructure with AMD hardware, and identified that:
1. The error does not occur when running directly on the machine
2. The error does occur when running in docker bound with the
3. The error is resolved by adding
I'll attach the code I used for testing in a reply.
It'd be great to have AMD pods use a more permissive security profile to improve AMD GPU support. Let me know if I can help with this in any way.
In particular, pinned memory allocation fails with a misleading
Error: Failed to allocate pinned memory: out of memory (2)I was able to do some testing on compute infrastructure with AMD hardware, and identified that:
1. The error does not occur when running directly on the machine
2. The error does occur when running in docker bound with the
docker run --device /dev/kfd --device /dev/dri/renderDXXX ...3. The error is resolved by adding
--security-opt seccomp=unconfinedI'll attach the code I used for testing in a reply.
It'd be great to have AMD pods use a more permissive security profile to improve AMD GPU support. Let me know if I can help with this in any way.
