Runpod•2y ago

[Urgent] One GPU suddenly went away

Hi, we have prod issue right now one of the gpu from our pod suddently disappared

Superintendent•1/13/24, 12:33 AM

fall off the bus?

ErcanOP•1/13/24, 12:34 AM

@Justin Can someone help, and check our pod ?

Superintendent•1/13/24, 12:35 AM

lspci | grep VGA

Superintendent•1/13/24, 12:35 AM

should spit out something about the gpu

EErcan Hi, we have prod issue right now one of the gpu from our pod suddently disappare...

Superintendent•1/13/24, 12:37 AM

also can u run nvidia-smi and show it

SSuperintendent also can u run nvidia-smi and show it

ErcanOP•1/13/24, 12:39 AM

yes, here first gpu got missing

Superintendent•1/13/24, 12:39 AM

so what does
lspci | grep VGAlspci | grep VGA spit out?

ErcanOP•1/13/24, 12:42 AM

cant run it, command not found

ErcanOP•1/13/24, 12:42 AM

not sure what to install

SSuperintendent so what does `lspci | grep VGA` spit out?

ErcanOP•1/13/24, 12:42 AM

what to install ?

Superintendent•1/13/24, 12:42 AM

wdym

Superintendent•1/13/24, 12:42 AM

lspci isnt there?

ErcanOP•1/13/24, 12:42 AM

Superintendent•1/13/24, 12:42 AM

run lspci

Superintendent•1/13/24, 12:42 AM

without grep

ErcanOP•1/13/24, 12:43 AM

Superintendent•1/13/24, 12:43 AM

lspci

Superintendent•1/13/24, 12:43 AM

your missing an i

Superintendent•1/13/24, 12:43 AM

lspcilspci

ErcanOP•1/13/24, 12:43 AM

Superintendent•1/13/24, 12:43 AM

whar.

ErcanOP•1/13/24, 12:43 AM

is not this for amd ?

Superintendent•1/13/24, 12:43 AM

wdym

Superintendent•1/13/24, 12:43 AM

your on nvidia gpus

Superintendent•1/13/24, 12:43 AM

so yea it should work

ErcanOP•1/13/24, 12:44 AM

yea that is what I am double checking if this should work with nvdia

Superintendent•1/13/24, 12:44 AM

sudo apt-get update
sudo apt-get install pciutils

Superintendent•1/13/24, 12:44 AM

try that

Superintendent•1/13/24, 12:44 AM

i think its something to do with pciutils

ErcanOP•1/13/24, 12:44 AM

ErcanOP•1/13/24, 12:44 AM

worked now

Superintendent•1/13/24, 12:45 AM

what does

dmesg

dmesg

spit out

ErcanOP•1/13/24, 12:45 AM

dmesg: read kernel buffer failed: Operation not permitted

Superintendent•1/13/24, 12:46 AM

sudo !!

Superintendent•1/13/24, 12:46 AM

(Wtf?)

ErcanOP•1/13/24, 12:46 AM

cant run sudo on Pods

Superintendent•1/13/24, 12:46 AM

oh right

Superintendent•1/13/24, 12:46 AM

docker container

Superintendent•1/13/24, 12:46 AM

uhh.

ErcanOP•1/13/24, 12:47 AM

yea

Superintendent•1/13/24, 12:47 AM

i have no friggin clue, can u try to restart it?

ErcanOP•1/13/24, 12:47 AM

will try but moved all production process to the running gpu

ErcanOP•1/13/24, 12:48 AM

so if I restart need to run bunch of things again

ErcanOP•1/13/24, 12:48 AM

I am just waiting maybe it comes back

EErcan will try but moved all production process to the running gpu

Superintendent•1/13/24, 12:48 AM

oof is the 4090 ok with the load?

ErcanOP•1/13/24, 12:49 AM

we have a internal queue, set it to 1 right now

ErcanOP•1/13/24, 12:49 AM

also just realized this is community cloud

Superintendent•1/13/24, 12:49 AM

yes

ErcanOP•1/13/24, 12:49 AM

thought this was on secure cloud

[Urgent] One GPU suddenly went away

Similar Threads

Similar Threads

Similar Threads