loading container image from cache
Since yesterday, and now on two different DC's, loading from cache just doesn't work.
I spent over an hour yesterday trying to load a ~4gb image from cache. A couple pods I let run for over 20 minutes before terminating. I eventually gave up on US-IL-1 where my storage was setup, and started one up on US-CA-2, which went fine as it pulled from the repo instead. I was completely loaded up, including redownloading about 30gb of models, while a pod I had left starting in IL was still "loading container image from cache."
I started up a pod on US-CA-2 at 10:08am PSD and right now it's 10:31am PSD, and still "loading container image from cache."
When I loaded from the repo, it had the image in 5-10 minutes last night. Any way I can opt-out of using the cache?
6 Replies
here's the current pod id: dtubgojwg9fcq6
Interesting, one sec.
I've escalated this to our infrastructure team, I don't want to ring the alarm bells until another user reports a reproduction as I don't have access to the logs at the moment.
oh wow, after 50 minutes it finally loaded. Not exactly happy about that
We disable the model cache on the machine your Pod is deployed to.
But we know what the issue is and it's being worked on so the model cache will be enabled after the fix and you should be good to go.
thank you
Update, we expect this fix to land today