Serverless fails when workers arent manually set to active

As the title says, my requests to my serverless endpoint are retrying/failing at a much higher frequency when my workers arent set to active. Anyone experienced something like this before?