Can I select the GPU type based on the base model in python script ?
Some base models can run on a 24G GPU, while others require a 48G GPU.
7 Replies
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
My requirement is to generate photos in different styles using different base models.
Some base models can complete the task with a 24G GPU, while others require a 48G GPU (24G would cause memory overflow).
I will pass the base model name as a parameter. Currently, the code only returns CPU, GPU, and XPU.
device = get_torch_device() generator = torch.Generator(device=device).manual_seed(seed)
Can I specify the GPU specifications?
For example, if my base model name is x, I want to add a condition in the code like if(base_model == x) gputype = 48G else gputype = 24G。
Nope, not possible, GPU configuration is at endpoint level, not request level
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
Yep thats the only solution
I see， thanks
This is RunPod adjecent, but I came across this library:
https://github.com/anyscale/llm-router
This seems to be what you're looking for; however, you'd still have to do a lot of custom configuration to get this running and I believe the above guidance still applies.