R
RunPoddaemon

Profiling CUDA kernels in runpod

Hi! I'm trying to profile my kernel with nsight-compute and I'm getting error : "==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0." Which is explained on this page : https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters and has to be fixed on the host side. Anybody found a workaround for this issue or how to solve it? Thanks!
M
Madiator201118d ago
you cant do it on RunPod as containers are not provilaged and exposing --cap-add=SYS_ADMIN would cause security risk
D
daemon17d ago
Okay, Thanks for the response. Any idea where can I do this?
M
Madiator201117d ago
im not sure what you trying to do
D
daemon17d ago
I'm actually trying to profile few of my custom CUDA kernels running on NVIDIA GPUs
M
Madiator201117d ago
you probably wont be able to do it from container
Want results from more Discord servers?
Add your server
More Posts
Inconsistency with volumesWe have an issue where when we startup a container/pod we run a script that should exists inside of Bug prevents changing a Serverless Pod to a GPU Podhttps://i.imgur.com/DNxVc1y.gifNo availability issueWhen renting some instances, the main screen says 'High availability', or etc.. yet it has none whenError: CUDA error: CUDA-capable device(s) is/are busy or unavailableI have 15 production endpoints deployed using Runpod and today they started to raise this error randL40 and shared storageFor my workloads I want to use a L40, but I also need shared storage. Do I get it right, that this iRun container only onceHi everyone, I want to run a container for a single life-cycle only (i.e. my container is designed tAuto-scaling issues with A1111Hey, I'm running an A1111 worker (https://github.com/ashleykleynhans/runpod-worker-a1111) on ServerlClone a Runpod NetworkvolumeHi! Is there some way to clone a Network Volume in the Runpod interface or is this something i have Insufficient Permissions for Nvidia Multi-GPU Instance (MIG)I was planning to test some new Nvidia GPU features using a pod with Nvidia A100 80G. I tried `nvidAutomatic1111 - Thread creation failed: Resource temporarily unavailableHello, we started to get this error more often. Normally we were getting it time to time, and after How can I view logs remotely?Hi! I am ttrying to view the logs of a training build I am doing but it seems to stop here. The contHow to make Supir in Serverless?Please tell me how to create serverles with the supir project? Or perhaps someone can do this for moCan we use serverless faster Whisper for local audio?I deployed faster Whisper using serverless and invoked it using "import requests url = "https://apis there any method to deploy bert architecture models serverlessly?is there any method to deploy bert architecture models serverlessly?