Do you cache docker layers to avoid repulling ?
I have a question, does the current serverless system caches the docker layers to avoid repulling ?
Let's assume I make a docker image with the first layers being the AI models, if I upload this, the dokcer repo will only get the new ones and won't update layers it already knows
If I pull the same will happen
Does the Serverless docker pulling also does the same? so that if I update my image, it will avoid repulling the layers it already knows ?
7 Replies
Unknown User•8mo ago
Message Not Public
Sign In & Join Server To View
As far as I know, some data centres do, and some don't. For example, on EU-RO I had to redownload the whole image every time. On EU-CZ it's pulling only the difference.
Unknown User•8mo ago
Message Not Public
Sign In & Join Server To View
EU-RO DC is bigger and has lot of machines so there is higher chance you end up on new machine that has no image cached
- all data centers support local layer caching
- some data centers with network storage also offer caching in network storage for serverless
- local cache is always used if the changes you make to endpoint don't change gpu, cuda version or some other attributes which require changing the server, otherwise we keep the worker and have it update in-place
I would check EU-RO then. I've never experienced layer caching there. I didn't even know it's supported on RunPod until I was able to use EU-CZ and was pleasantly surprised it's there. This was happening even on a tiny few kb rewrites of the handler file. While cached just fine by the docker hub on upload, the data centre always pulled the whole huge image again. It's wasting your bandwidth there for sure.
I keep seeing it over and over again - I change a couple of topmost lightweight layers yet workers re-downloads 10GB+. Is there anything I can do with my image build process to avoid it?