Disk reading unacceptably and mind boggingly slow
I thought I had figured out where to put my data:
pay for extra disk space in /
move the data from /workspace (which is a network drive) to / and from there it can be read fast enough.
But today, I tried the same thing in this pod:
RunPod Pytorch 2.1 ID: yuicz8xifys47n 1 x A100 SXM 80GB 16 vCPU 250 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 and the reading speed from / was as horrible as reading from /workspace. the pod is down now cause obviously i'm not keen to pay for useless machinery So what are we supposed to do I don't understand how anyone can do ML like this.
RunPod Pytorch 2.1 ID: yuicz8xifys47n 1 x A100 SXM 80GB 16 vCPU 250 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 and the reading speed from / was as horrible as reading from /workspace. the pod is down now cause obviously i'm not keen to pay for useless machinery So what are we supposed to do I don't understand how anyone can do ML like this.
3 Replies
Network drive is garbage, I agree. Someone else had this issue, and I think the way they solved it is they use network drive as a storage area, and then copy out into the container disk area, and they found substantially sped up their workflow
Hi Justin, I'm pretty sure I am that someone since I remember chatting with you!
And the problem is that yesterday even though I copied my data to the container disk area ( at the root of the filesystem if I'm not very mistaken: "/" ), the reading was just as slow as from the network disk.!!
Same problem today
RunPod Pytorch 2.1
ID: 5vzts37ixeqaz7
1 x A100 80GB
8 vCPU 188 GB RAM
runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04
On-Demand - Secure Cloud
My data is in /scratch
and the script that just reads records from the database one by one works at 2 iterations/second.
On a computer where the .db is read from an SSD this runs at 600 iterations/second.
So i'm pretty sure there's pods where even / is on the network ?
@Papa Madiator I assume this is what he is referring to