panos.firbas Comments - Answer Overflow

Topics

panos.firbas

•Created by panos.firbas on 2/29/2024 in #⛅｜pods-clusters

Disk reading unacceptably and mind boggingly slow

Same problem today RunPod Pytorch 2.1 ID: 5vzts37ixeqaz7 1 x A100 80GB 8 vCPU 188 GB RAM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 On-Demand - Secure Cloud My data is in /scratch and the script that just reads records from the database one by one works at 2 iterations/second. On a computer where the .db is read from an SSD this runs at 600 iterations/second. So i'm pretty sure there's pods where even / is on the network ?

6 replies

•Created by panos.firbas on 2/29/2024 in #⛅｜pods-clusters

Disk reading unacceptably and mind boggingly slow

Hi Justin, I'm pretty sure I am that someone since I remember chatting with you! And the problem is that yesterday even though I copied my data to the container disk area ( at the root of the filesystem if I'm not very mistaken: "/" ), the reading was just as slow as from the network disk.!!

6 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

I'm pretraining now

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

it makes some sense that one is faster the other is bigger

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

looks like 3090 has more 'cores'

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

hehe A100 80GB PCIe

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

it's still a little slower in the a100 than in my 3090 for a small model, is that to be expected?

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

So yeah, i'm running it now with the data in / and it's going fast so that's the solution

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

Thanks a lot !!

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

although that move SHOULD be fast

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

yeap

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

work there, then move it out

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

in /scratch or something

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

i spawn with big container, move the data there

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

oke that's the solution then

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

ah the container disk option ?

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

but i cant both have a network image AND lots of space in / can i

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

yeah maybe i just stuff it in the docker image

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

more like 500k files actually

142 replies

•Created by panos.firbas on 2/16/2024 in #⛅｜pods-clusters

How should I store/load my data for network storage?

i don't think 100k or so files would make the filesystem happy

142 replies