downloading containers takes ages
I dont know how in other regions but in EU-RO-1 it takes even an hour today to download a container from dockerhub, I restarted and created completly new pods, no difference
4 Replies
I wanted to test training on B200 but it will cost more to download container than use it 🤣
Docker Hub is super annoying about some ratelimits and usage based throttling, we can't really control it and they have not been receptive to our requests for partnership related lifts of these limits (which is a program that exists!)
It is hard to say if it's a rate limiting issue on Dockerhub, but one way you can test is to add a registry authentication under setting, and when you launch your template try to select that registry authentication credentials
I tested EU-RO-1 myself with a pytorch template official + a custom template (trying to emulate the pull) and worked fine for me. But I can see this being frustrating espsecially if you want to do fine tuning jobs
If there is no need with a custom template, lauching on a runpod official template, generally should be the fastest as we keep those cached downloads as much as possible
I have custom template and use creds to priv repo on dockerhub, so maybe I should use some other container repo (but I use dh's cloud builds which are convinient...), uploading build was fast so I thought it is on runpod's side