how to host 20gb models + fastapi code on serverless
I have 20gb model files and a fastapi pipeline code to perform preprocessing and inference+ training.
How can I use runpods serverless?
14 Replies
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
It's Pytorch + tensorflow
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Do I have to dockerize the models with code?..
The docker image is around 50GB
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah.. that's why thought to keep model out of docker
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Perfect that sounds a plan
Thanks
Can u share links for the network storage access and deploy too plz
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Don't use fastAPI on serverless, its already an API layer.
and don't put your models in your image most likely, def use network drives
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
just several reasons to prefer smaller images, it will work but with a lot of overhead
Network drives are about 1 million percent slower than baking things into the image, so I don't know why you are saying its better, because you are wrong.
Its always better to bake a model into the image wherever possible every single time, you should only use the garbage network storage if you absoultely have to.