Network Storage Performance Issue - 4.8x Slowdown
I'm experiencing severe performance degradation with network storage (MooseFS/FUSE) on euro-2 region.
The Problem:
My Python application (ComfyUI with many custom nodes) takes 14min 26sec to start on RunPod vs 3 minutes on my local machine with identical setup.
Root Cause (Verified with benchmarks):
Network storage latency on small file operations:
- Small file writes: 41.35ms per file (vs 0.1ms on local SSD) = 400x slower
- Metadata ops (stat): 12.59ms per call (vs 0.1ms local) = 125x slower
- Directory scan: 20.8 seconds to list 2,016 directories recursively
Impact:
My app loads 2,808 Python files during startup. With 41ms per file, that's ~115 seconds just in I/O latency (not counting actual processing).
Sequential I/O is great (438 MB/s), but the cumulative latency on thousands of small files kills performance.
This makes RunPod challenging for Python apps with many dependencies/modules. Would love to find a solution!
Hardware:
- Region: euro-2
- GPU: RTX 2000 Ada
- Mount:
Happy to provide detailed diagnostic report if needed.
The Problem:
My Python application (ComfyUI with many custom nodes) takes 14min 26sec to start on RunPod vs 3 minutes on my local machine with identical setup.
Root Cause (Verified with benchmarks):
Network storage latency on small file operations:
- Small file writes: 41.35ms per file (vs 0.1ms on local SSD) = 400x slower
- Metadata ops (stat): 12.59ms per call (vs 0.1ms local) = 125x slower
- Directory scan: 20.8 seconds to list 2,016 directories recursively
Impact:
My app loads 2,808 Python files during startup. With 41ms per file, that's ~115 seconds just in I/O latency (not counting actual processing).
Sequential I/O is great (438 MB/s), but the cumulative latency on thousands of small files kills performance.
This makes RunPod challenging for Python apps with many dependencies/modules. Would love to find a solution!
Hardware:
- Region: euro-2
- GPU: RTX 2000 Ada
- Mount:
mfs#euro-2.runpod.net:9421 type fuseHappy to provide detailed diagnostic report if needed.
