Getting slow workers randomly
we’re running a custom comfy ui workflow on RTX 4090 instances with a volume attached to them. Around 70% of the time, we get normal workers where the delay time is around 8-10 seconds.
But sometimes we get random slow workers. I attached a screenshot of the requests logs, you can see that the worker with
When we get a slow worker, it's just slower on every aspect. GPU inference takes 5x longer. Comfy import times take 7-8x longer than a normal worker.
But sometimes we get random slow workers. I attached a screenshot of the requests logs, you can see that the worker with
hv5rbk09kzckc9 id takes around 11-12 seconds to execute the same exact comfy workflow with same gpu whereas the other worker with id lgmvs58602xe61 takes 2-3 seconds to execute.When we get a slow worker, it's just slower on every aspect. GPU inference takes 5x longer. Comfy import times take 7-8x longer than a normal worker.
