Unstable speed of processing between different wroker.
Hi! I'm deploying serverless for model SadTalker on endpoint with specs 24GB GPU Pro. And I tested some requests and realized that amount of processing time with the same request on different workers are huge difference. Here are 2 log files:
1 - Log of slower worker: it take 45s executionTime. spead of iteration is 2.09s/it at Face Render process
2- Log of normal worker: it take 21s executionTime. Speed of iteration is approximate 1.30 it/s at Face Render process
My endpoints is:schx1xwzhn1lhk
Could anyone help me to debug and prevent this issue?
1 - Log of slower worker: it take 45s executionTime. spead of iteration is 2.09s/it at Face Render process
2- Log of normal worker: it take 21s executionTime. Speed of iteration is approximate 1.30 it/s at Face Render process
My endpoints is:schx1xwzhn1lhk
Could anyone help me to debug and prevent this issue?
