Topics

RunPod•3mo ago

P2P is disabled between NVLINK connected GPUs 1 and 0

Hey team! Could you fix NVLink issue for H100 SXM Community pods? I encounter this error frequently. Corrupted pod ID: 4a5acwxj2kene6 P2P is disabled between NVLINK connected GPUs 1 and 0. This should not be the case given their connectivity, and is probably due to a hardware issue. If you still want to proceed, you can set NCCL_IGNORE_DISABLED_P2P=1. I can proceed with NCCL_IGNORE_DISABLED_P2P flag but this will drop performance ~ 10%

No description

Solution:

@storuky2306 so got response and aparently gpu5 is not supporting P2P. What we can advise for now is to pick diffrent machine...

Jump to solution

3 Replies

Madiator2011•3mo ago

forwarded it to team

Solution

Madiator2011•3mo ago

@storuky2306 so got response and aparently gpu5 is not supporting P2P. What we can advise for now is to pick diffrent machine

storuky2306•3mo ago

@Papa Madiator ok, thanks

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

11KMembers

View on Discord

Want results from more Discord servers?

Add your server

More Posts

Pod with different IPS?Hi team, how do I create 2 pod with different IP address? I have 2 right now but they have the same No GPU AvailableHello, I have using the 'runpodctl' workflow with dev and deploy commands and I have one network vol Find Config of Deleted PodIs there any way to view the *config* of a pod that was already deleted? I see the pod creation lis How to load model into memory before the first run of a pod?In the template worker, in the handler file it is written: ```py # If your handler runs inference on Increase number workersI have at the moment a number of workers of 10. How can I increase the maximum number of them ?High execution time, high amount of failed jobs36% of my serverless jobs have failed this morning and the execution time has spiked massively. Thi How do I write handler for /runWhen I create a handler `runpod.serverless.start({"handler": async_generator_handler})` Only http://How do indicated job status in a handler?For example in https://docs.runpod.io/serverless/workers/handlers/handler-async ``` import runpod im A6000 serverless worker is failing for an unknown reason.In the last week a few of our serverless workers have been failing on all requests. Trying to narrow Can multiple models be queried using the vllm serverless worker?Just getting started with the vllm serverless worker and my first question is can I query multiple m