Slow Disk Performance For H200 nodes
"Container volumes provide fast read and write speeds since they are locally attached to workers." Based on the RunPod documentation, I would expect there to be local SSDs with the pods. However, I've noticed getting slow disk speeds when I'm using certain H200 nodes. I'd expect 5GB/s+ with local NVMe SSDs, but I'm getting < 1 GB/s with some pods. Here's the command I use to benchmark:
My company is saving very large model checkpoints, so we need performant disk speeds.
fio --name=write-test --directory=. --numjobs=8 --size=10G --rw=write --bs=128K --iodepth=1 --time_based --runtime=10s --group_reporting --direct=1My company is saving very large model checkpoints, so we need performant disk speeds.
