Load Balancer Endpoint - "No Workers Available"
I tried using Load Balancer Endpoint today. I got it to work successfully as well. However, after successful uses. I noticed some annoying behavior.
- When testing, you're going to have lots of test runs, so you'll have to hit the endpoints multiple times. However, once you reached the point of, let's say, the 12th time of sending the task to the worker, it's gonna return "no workers available" despite having it but in "Idle" state.
- When doing inference (for context, I use LatentSync. So it takes about a good 2-5 mins), I had to manually hit
/pingin order to prevent the worker to become "Idle", which is kind of annoying.
