Profiling CUDA kernels in runpod
Hi! I'm trying to profile my kernel with nsight-compute and I'm getting error : "==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0."
Which is explained on this page : https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters
and has to be fixed on the host side. Anybody found a workaround for this issue or how to solve it? Thanks!
5 Replies
you cant do it on RunPod as containers are not provilaged and exposing --cap-add=SYS_ADMIN would cause security risk
Okay, Thanks for the response. Any idea where can I do this?
im not sure what you trying to do
I'm actually trying to profile few of my custom CUDA kernels running on NVIDIA GPUs
you probably wont be able to do it from container