Questions on large LLM hosting
1
I see mentions of keeping a model in a Network Volume to share between all endpoints. But if I already have my model inside of a container image-- wouldn't my model already be cached in that image? Which would be faster for cold boots?
2...
serverless container disk storage size vs network volume
When I add a Serverless endpoint, it defaults to 5GB container disk. I tried to change it to a crazy high number like 50000GB, and it seems to be ok with it??
I'm confused, is this disk storage physically attached to the GPU machine? there a limit of this storage size? Does it cost any extra money? My ComfyUI docker image needs to download many different models, which can be hundreds of GB in total, what would happen if it exceeds the storage limit?
If I choose to attach a network volume, does it mean my docker image (contains many different models) will be deploy to store in the network volume? And the deployed volume need to communicate with the GPU via network requests?? So it will be slower? ...
401 Unauthorized
having used a combination of ashleyk's readme and generativelabs video, I have setup a runpod oobabooga worker. However I am unable to make calls to the endpoint resulting in a 401 unnauthorized:
I have setup postman as the generativelabs video has suggested, and I have quadrouple checked that I have properly setup the api id & api key on postman and SAVED.
I am also using the format as per @ashleyk's documentation (in fact i'm using the exact prompt), however I am still getting 401 unauthorized. As such i'm at a loss. Hope iI can get help...
Solution:
I ended up solving it by puting the API Key under Authorization as Type bearer instead of the header tab, and by manually putting the id into the URL
Billing increases last two days heavily from delay time in RTX 4000 Ada
I checked my billing history and I saw that my serverless bills increased a lot and the culprit is the usage of RTX 4000 Ada.
I checked the log and it's because of CUDA runtime error as ONNX runtime doesn't work with this specific GPU. This causes the container to retry again and again, increasing the delay time.
I have never met this issue before so I'm not sure why it's happening now without any code change on my side.
...
