Network Volume Utilization
I'm using RunPod to train my model. And I use RunPod network volume to store my artifacts.
I created a volume with 60GB, and it's path is /workspace. When I finish uploading my data, it says the network volume is already at 80% utilization. While when I do du -sh /workspace/, it only shows 17GB, (which is what I expected), but why I'm seeing 80% utilization, do I need to do some cleaning, after unzip my data?
The volume is on EU-RO-1
5 Replies
One thing to add, is that while unzipping my data, I found out the original network volume might not be enough (20GB originally), so I increase it while unzipping the data, not sure if this creates any duplicates in the network volume.
Unknown User•5mo ago
Message Not Public
Sign In & Join Server To View
Do you happen to have a ton of small files
Like a million text files or images
For the training data
Unknown User•5mo ago
Message Not Public
Sign In & Join Server To View
I experienced that
sooooo
🥲
turns out metadata for millions of text files were large
no it was like 8million i think
total file count was 10M+