is it possible to send request to a specific workerId in a serverless endpoint?
I need to have a custom logic to distribute requests to available workers in the serverless endpoint. Is there a way to send request to a specific worker using workerId?
7 Replies
this feature is not available, is it because you have a specific model loaded on the worker id?
yes, exactly. I though of tracking particular workerId models and route the requests accordingly
if you have any other solution for this, please let me know. I am using comfy btw
we are brainstorming around this use case, dont have any eta yet, likely goal is to allow some type of routing in future
cool, thanks
@flash-singh Any updates regarding this issue? If we are able to send requests to specific worker after getting list of available workers via API then that will be awesome.
Thanks
I think you have several options:
- Optimise your image and code to the point where model loading and switching are not costly operations. (That's most likely the best choice.)
- If you don't have a large number of models, use a separate endpoint for each one.
- Or use a bigger GPU and keep more of them loaded in the VRAM.
Use separate endpoints for each model.
That is really the best way to go