Runpod storage configuration
Hello, I am working with a dataset off 4 mill files. Normally it takes up about 23gb on my pc. But on runpod its about 10-15x that size. The reason I have found is that runpod takes up 100kb for a file. My files are each around 6-10 kb, but ubuntu which pod is running from is configured to allocate 100kb min. for a file. I don't have permission within the pod to manage that. What can I do about this?
5 Replies
Unknown User•10mo ago
Message Not Public
Sign In & Join Server To View
For our file system, having a lot of small files can create extra overhead and impact disk usage
sqlite is good
Unknown User•9mo ago
Message Not Public
Sign In & Join Server To View
this will depend on your use case, you can store tar file in network storage, and then extract in local container disk for actual work, that way network storage is looking at 1 or few files instead of millions