What is expected continuous delivery (CD) setup for serverless endpoints for private models?
Hello, our model artificats are stored in S3, what is the continuous delivery setup for serverless models not hosted on dockerhub?
What I have seen so far:
- Existing runpod workers download publicly available models and push them to dockerhub
- Github repo connection in Serverless setup, I'm not sure how I will pass my AWS credentials during the runpod managed build to download the model
- Network volumes that can be attached to serverless. I see cloudsync works runpod -> S3 not the other way around. How would we programmatically update this volume and refresh our model?
- No native auth integration with AWS ECR, credentials expire in 12 hours, which effects reloading containers
What I have setup right now:
- Github action builds and uploads AWS ECR image
- Manually update credentials
- change tag version for new release (i'm okay with doing this manually if I have to for now)
3 Replies
Unknown User•9mo ago
Message Not Public
Sign In & Join Server To View
for a quick win we can get you programmatic way to update creds, along with updating tag version
our long term path is we are introducing
model store which can pull public and private models from huggingface and store them locally on servers with faster access rather than in network storage; s3 support may be further down the roadSo I just added a cron schedule i.e cloudwatch event -> lambda to update credentials...
would be nice to get good documentation on best practices on deploying private models if you are in AWS setup