Worker hangs for really long time, performance is not close to what it should be
Hi, I'm working with a transcription and diarization endpoint. The docker image works great, tested locally and also inside a worker, I ssh into the worker and tested using:
The processing time is around 1 minute for this video (11 min), works great, these are the logs I get from running inside the worker the same reques -> Message.txt appended.
Once I test this endpoint using a normal request the worker behaves completely abnormal, taking more than 5-6 minutes just to start the transcription, then even more minutes transcribing. The really weird part is I tested the handler in the worker itself using ssh, I have no idea how to debug this or what might be happening:
This should happen in matter of seconds, just liek the logs from the execution within the worker says:
Manual execution
Weirdly enough, I have another endpoint using just transcription, its also fast:
The processing time is around 1 minute for this video (11 min), works great, these are the logs I get from running inside the worker the same reques -> Message.txt appended.
Once I test this endpoint using a normal request the worker behaves completely abnormal, taking more than 5-6 minutes just to start the transcription, then even more minutes transcribing. The really weird part is I tested the handler in the worker itself using ssh, I have no idea how to debug this or what might be happening:
This should happen in matter of seconds, just liek the logs from the execution within the worker says:
Manual execution
Weirdly enough, I have another endpoint using just transcription, its also fast:
Solution
Take a look at our implementation of Fast Whisper https://github.com/runpod-workers/worker-faster_whisper/blob/main/src/rp_handler.py
Your code is already blocking, async is likely just introducing complexities
Your code is already blocking, async is likely just introducing complexities
