Intermittent timeouts on requests

I have a custom docker image serverless endpoint. I am sending a payload with the python package endpoint.run_sync(payload, timeout=60). I currently have 0 active workers. I can typically send the first request. After completion, if I send a following request before that worker times out, it will often timeout without ever logging that the main function started. Basically, it seems like the message never gets in the RunPod Queue. What could be causing this behavior? How can I avoid it (or debug it)?
Logs are attached - this case is 2 successful requests, then a third request just times out - it seems like the request never gets to the queue (no logs).