Unhealthy machines

Hhotsnr We recently noticed that occasionally we get machines with bad performance - wor...

P

PoddyAPP•6/18/25, 1:00 AM

@hotsnr

Escalated To Zendesk

The thread has been escalated to Zendesk!

J

Jason•6/18/25, 1:01 AM

please open a ticket, i wonder how do you check if the worker has bad performance?

D

Dj•6/18/25, 4:39 PM

Support may tell you that you can just trigger a refresh of the worker with its ID which I recommend, if it doesn't fix your issue then you have something else going on.

Hhotsnr We recently noticed that occasionally we get machines with bad performance - wor...

H

Henky!!•6/18/25, 5:51 PM

What does crashing accomplish?

JJason please open a ticket, i wonder how do you check if the worker has bad performanc...

H

hotsnrOP•6/19/25, 1:03 AM

Our system generates audio in real time, and users complained about lags. Logs indicate that Real-time ratio (RTM) was way higher than usual.

DDj Support may tell you that you can just trigger a refresh of the worker with its ...

H

hotsnrOP•6/19/25, 1:09 AM

This was a freshly started worker, my understanding is that refresh basically restarts container and doesn't recycle/move it to another host.

J

Jason•6/19/25, 1:09 AM

Whats real time ratio?

HHenky!!What does crashing accomplish?

H

hotsnrOP•6/19/25, 1:09 AM

Job won't be able to land on this worker, and user wouldn't suffer.

JJason Whats real time ratio?

H

hotsnrOP•6/19/25, 1:12 AM

Sorry, it's Real-time factor (RTF). If you generate X seconds of audio and it took your system Y seconds to do it, your RTF is Y/X. Target is to have it <1 so that you can keep up with playback.

H

hotsnrOP•6/19/25, 1:12 AM

We also noticed that models took much longer too load for that worker.

J

Jason•6/19/25, 1:13 AM

how do you check if the worker has bad performance?

long model loading times is it compared to other workers?

JJason how do you check if the worker has bad performance? * long model loading times ...

H

hotsnrOP•6/19/25, 1:16 AM

Yes, by comparing to other workers - all of them were of the same type (4090) in the same DC (IL-1).

H

hotsnrOP•6/19/25, 1:20 AM

I'm going to file a support request once I get more data from our logging system. I wanted to understand whether I can do something better than 1/2 (we can't allow jobs to land on HW which is clearly bad).

J

Jason•6/19/25, 1:26 AM

Maybe take whats taking long, it may be the
queue delay time, worker startup( which is included in queue delay, this includes model loading)
or inference time takes too long relative to other worker (slow gpu processing, networking for upload, or disk speed for reading / saving result perhaps)

Missing signals to confirm:
–
```
nvidia-smi -q -d CLOCK,POWER,PERF
```
nvidia-smi -q -d CLOCK,POWER,PERF
```
nvidia-smi -q -d CLOCK,POWER,PERF
```
nvidia-smi -q -d CLOCK,POWER,PERF
from good vs. bad pod
– Disk throughput during model load:
```
fio
```
fio
```
fio
```
fio
or
```
dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=direct
```
dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=direct
```
dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=direct
```
dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=direct
– CPU freq scaling:
```
lscpu | grep MHz
```
lscpu | grep MHz
```
lscpu | grep MHz
```
lscpu | grep MHz
,
```
cpupower frequency-info
```
cpupower frequency-info
```
cpupower frequency-info
```
cpupower frequency-info
– Exact timestamps for queue delay vs. container boot vs. model-load vs. first inference

(if you want)

Immediate Steps (while incident is active)
– Quick synthetic benchmark at container start; exit ≠ 0 if below threshold so RunPod restarts it automatically.
– Capture the four diagnostics listed above from a “bad” pod and attach them to the Zendesk ticket

# healthcheck.sh  (exit code 2 => pod considered unhealthy)
python - <<'PY'
import os, time, torch, json, sys
torch.manual_seed(0)
start=time.time()
a=torch.rand(4096, 4096, device='cuda')
for _ in range(256):
  _=(a@a).sum()
torch.cuda.synchronize()
elapsed=time.time()-start
th=float(os.getenv("RTF_THRESHOLD", "3.0"))
print(json.dumps({"startup_benchmark_sec": elapsed}))
sys.exit(2 if elapsed>th else 0) # or however you crashed it
PY

# healthcheck.sh  (exit code 2 => pod considered unhealthy)
python - <<'PY'
import os, time, torch, json, sys
torch.manual_seed(0)
start=time.time()
a=torch.rand(4096, 4096, device='cuda')
for _ in range(256):
  _=(a@a).sum()
torch.cuda.synchronize()
elapsed=time.time()-start
th=float(os.getenv("RTF_THRESHOLD", "3.0"))
print(json.dumps({"startup_benchmark_sec": elapsed}))
sys.exit(2 if elapsed>th else 0) # or however you crashed it
PY

# healthcheck.sh  (exit code 2 => pod considered unhealthy)
python - <<'PY'
import os, time, torch, json, sys
torch.manual_seed(0)
start=time.time()
a=torch.rand(4096, 4096, device='cuda')
for _ in range(256):
  _=(a@a).sum()
torch.cuda.synchronize()
elapsed=time.time()-start
th=float(os.getenv("RTF_THRESHOLD", "3.0"))
print(json.dumps({"startup_benchmark_sec": elapsed}))
sys.exit(2 if elapsed>th else 0) # or however you crashed it
PY

# healthcheck.sh  (exit code 2 => pod considered unhealthy)
python - <<'PY'
import os, time, torch, json, sys
torch.manual_seed(0)
start=time.time()
a=torch.rand(4096, 4096, device='cuda')
for _ in range(256):
  _=(a@a).sum()
torch.cuda.synchronize()
elapsed=time.time()-start
th=float(os.getenv("RTF_THRESHOLD", "3.0"))
print(json.dumps({"startup_benchmark_sec": elapsed}))
sys.exit(2 if elapsed>th else 0) # or however you crashed it
PY

Bake the benchmark above into your worker entrypoint and run it before the handler python file

Unhealthy machines

Similar Threads

Similar Threads

Similar Threads