Runpod•2y ago

network storage sooo slow

Hi, I'm new to runpod. I'm running a 5xH100 in US-KS2 with network storage in the same region. Loading the model (70B Llama) from storage is going to take 25 minutes. Is this normal for runpod? This normally takes seconds on other machines I've used.

Jason•8/3/24, 12:05 PM

woow 5x h100?

Jason•8/3/24, 12:05 PM

isn't that overkill?

Jason•8/3/24, 12:06 PM

anyways for the performance in my experience i've been able to load llama2 70b in around 5 minutes~ or so ( longest time )

Jason•8/3/24, 12:06 PM

maybe you can report the pod if its too slow

PaleBlueDotOP•8/3/24, 9:50 PM

5 mins is far more reasonable than 25. It's for training, hence the h100s. What do you train 70B with? Not sure runpod is going to fit the bill if the GPUs are sitting idle for 25mins on every run. Just wondered if runpod was having a bad day and I should give it a second chance.

Yasmin•8/4/24, 12:06 AM

One thing to try is to use the regular storage, and see if it makes a difference.
I had an issue before with the map() function on network storage and I was told here it is slower. I reported to the official support. So probably, you can do the same; hopefully, the issue get some attention.

briefPeach•8/8/24, 6:23 AM

Network storage is slow…I bake the models into the container now and it’s much much faster model loading

tim•8/9/24, 12:28 PM

@briefPeach can you please let me know in which datacenter you used the network-storage?

briefPeach•8/9/24, 1:14 PM

EU-IS I think

briefPeach•8/9/24, 1:14 PM

Is it faster in other areas

tim•8/9/24, 1:14 PM

Was there a point in time when you didn't had this probem in EU-IS?

network storage sooo slow

Similar Threads

Similar Threads

Similar Threads