Inconsistent delay time with generator worker

I am getting very inconsistent delay times when running serverless with a generator handler. I have the expected delay if there is cold-start but then once a worker has started I then still get delay times ranging from 1 - 30 seconds. I would expect once the worker is started for the delay time to be low and consistent. I have max workers 2 and scale on number of requests in the queue. What's going on here?