R
Runpod•3mo ago
lraciringer

H100 in US-KS-2 has high CPU latency

Hey, I just reserved an H100 in US-KS-2 and the instance is super slow. It has taken over ten minutes just to install Python dependencies. It's not just download speed (which is slow) but also CPU because after the downloads complete, processing the wheel installations is going very very slow. I was using A100 last week and this process went really fast. 
root@7f041151afc9:/workspace/stanford-cs336-lmfs-assignment1-basics# uv run wandb login
Using CPython 3.13.5
Creating virtual environment at: .venv
      Built cs336-basics @ file:///workspace/stanford-cs336-lmfs-assignment1-basics
░░░░░░░░░░░░░░░░░░░░ [0/76] Installing wheels...                                                                                                                             warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set export UV_LINK_MODE=copy or use --link-mode=copy to suppress this warning.
**Installed 76 packages in 9m 58s**
root@7f041151afc9:/workspace/stanford-cs336-lmfs-assignment1-basics# uv run wandb login
Using CPython 3.13.5
Creating virtual environment at: .venv
      Built cs336-basics @ file:///workspace/stanford-cs336-lmfs-assignment1-basics
░░░░░░░░░░░░░░░░░░░░ [0/76] Installing wheels...                                                                                                                             warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set export UV_LINK_MODE=copy or use --link-mode=copy to suppress this warning.
**Installed 76 packages in 9m 58s**
When I look at top on the instance, the CPU usage is negligible (everything at 0-2% CPU) during this process...
2 Replies
Dj
Dj•3mo ago
I had this experimenting with UV recently, it's a quirk of how they do caching and using network storage - you just need to change where the uv cache is so it's on the same file system as your network drive. https://github.com/astral-sh/uv/issues/7285#issuecomment-2343651950
GitHub
Failed to hardlink files: Issue with ruff cache · Issue #7285 · ...
When I run uv add or uv pip install I get this warning: ░░░░░░░░░░░░░░░░░░░░ [0/6] Installing wheels... warning: Failed to hardlink files; falling back to full copy. This may lead to degraded perfo...
lraciringer
lraciringerOP•3mo ago
Okay thanks I will check that. I saw similar performance issues with the Git checkout. (Slow to download a small repo, and then slow to process the repository after it was downloaded.) So I think there’s more to it than uv. I have seen the same uv warning on different RunPod machines but the commands completed quickly (<1 minute)

Did you find this page helpful?