Regarding the new serverless loadbalancing endpoint
I try it with my existing fastapi application and found the worker is unable to go back from running -> idle state. My application has asyncio processes running in the background. Is that the reason?
Also is the new loadbalancing endpoint support socket-based API?
Solution:Jump to solution
sockets should not work, i say should not because we use a proxy to forward the request, SSE will work, but doubt sockets will
11 Replies
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
Solution
sockets should not work, i say should not because we use a proxy to forward the request, SSE will work, but doubt sockets will
idle > running is determined by the health check, read into that, make sure the port is exposed properly in your container
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
running > idle is whenever the request is done and based on your idle timeout and min workers, e.g. if you have min worker to 1 then at least 1 worker will never idle
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
the serverless control plane doesn't care what's running in the worker, all it cares about for load balancing is that a request is sent to worker using http proxy and its handling it, once the request is done, it knows that no other requests are pending, then it follows idle timeout + min workers logic
so if you create a httpserver that takes a requests and runs a background job but returns the request and completes it, well the serverless doesn't not know of any background jobs, it considers it complete and then will stop the worker
long running requests beyond 10 mins are not a good use case for load balancer
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View
oh yes its that, we can increase it, 10 mins is likely the max we can get to in future
we may figure out a better pattern for long running jobs in future but queue serverless is the best solution for it
thanks @Jason @flash-singh well noted. I might experiment with a simple socket app to test.
Unknown User•2mo ago
Message Not Public
Sign In & Join Server To View