Disk reading unacceptably and mind boggingly slow

I thought I had figured out where to put my data:
pay for extra disk space in /
move the data from /workspace (which is a network drive) to / and from there it can be read fast enough.

But today, I tried the same thing in this pod:

RunPod Pytorch 2.1
ID: yuicz8xifys47n
1 x A100 SXM 80GB
16 vCPU 250 GB RAM
runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04

and the reading speed from / was as horrible as reading from /workspace.
the pod is down now cause obviously i'm not keen to pay for useless machinery

So what are we supposed to do I don't understand how anyone can do ML like this.

Ppanos.firbas I thought I had figured out where to put my data: pay for extra disk space in / ...

J.•2/29/24, 4:50 PM

Network drive is garbage, I agree. Someone else had this issue, and I think the way they solved it is they use network drive as a storage area, and then copy out into the container disk area, and they found substantially sped up their workflow

panos.firbasOP•3/1/24, 10:19 AM

Hi Justin, I'm pretty sure I am that someone since I remember chatting with you!
And the problem is that yesterday even though I copied my data to the container disk area ( at the root of the filesystem if I'm not very mistaken: "/" ), the reading was just as slow as from the network disk.!!

panos.firbasOP•3/1/24, 10:31 AM

Same problem today

RunPod Pytorch 2.1
ID: 5vzts37ixeqaz7
1 x A100 80GB
8 vCPU 188 GB RAM
runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04
On-Demand - Secure Cloud

My data is in /scratch

and the script that just reads records from the database one by one works at 2 iterations/second.
On a computer where the .db is read from an SSD this runs at 600 iterations/second.

So i'm pretty sure there's pods where even / is on the network ?

ashley•3/1/24, 11:03 AM

@Papa Madiator I assume this is what he is referring to

Disk reading unacceptably and mind boggingly slow

Similar Threads

Similar Threads

Similar Threads