Running Workers Not Shutting Down & Incurring Charges
Hi, we're facing a critical issue with workers not shutting down when there's nothing in queue/progress, which is causing significant over-billing and blocking our app launch. Reporting this after it happened at least 3 times.
I've observed that after all jobs are processed (finished/cancelled and nothing in queue), workers continue running for over 8 minutes doing nothing. I noticed it happening with both scaling settings:
- Queue Delay: A worker ran for 8+ minutes with an empty queue (attached a video of this below)
- Request Count: Two separate workers ran for 8+ minutes after the last job was processed (I sent these messages when it happened: https://discord.com/channels/912829806415085598/948767517332107274/1388527617510084651 https://discord.com/channels/912829806415085598/948767517332107274/1388531493768527932)
This costed me another $10 in credits over just two days. Just the two examples I shared above are examples where the workers run for 24 minutes (1 worker running over 8 mins on Queue Delay + 2 workers running over 8 mins on Request Count = 3 workers of RTX4090 GPU 24 GB (PRO) running over 24 mins) and charged us when it wasn't doing anything. I've spent $100 in just two months on testing alone, and issues like these are preventing me from launching our app since we can't rely on the platform scaling to function properly as we will launch the app in a server with over 10K members.
Everything on our app has been ready for two months, we have to launch as soon as possible when serverless endpoints work properly (please also see the other issue we have ⚡｜serverlessImage Generation Stuck Until New Requests Are Sent).
I'd really appreciate if you could help with this. I can share the logs with a DM if needed. Thank you for your time!
I've observed that after all jobs are processed (finished/cancelled and nothing in queue), workers continue running for over 8 minutes doing nothing. I noticed it happening with both scaling settings:
- Queue Delay: A worker ran for 8+ minutes with an empty queue (attached a video of this below)
- Request Count: Two separate workers ran for 8+ minutes after the last job was processed (I sent these messages when it happened: https://discord.com/channels/912829806415085598/948767517332107274/1388527617510084651 https://discord.com/channels/912829806415085598/948767517332107274/1388531493768527932)
This costed me another $10 in credits over just two days. Just the two examples I shared above are examples where the workers run for 24 minutes (1 worker running over 8 mins on Queue Delay + 2 workers running over 8 mins on Request Count = 3 workers of RTX4090 GPU 24 GB (PRO) running over 24 mins) and charged us when it wasn't doing anything. I've spent $100 in just two months on testing alone, and issues like these are preventing me from launching our app since we can't rely on the platform scaling to function properly as we will launch the app in a server with over 10K members.
Everything on our app has been ready for two months, we have to launch as soon as possible when serverless endpoints work properly (please also see the other issue we have ⚡｜serverlessImage Generation Stuck Until New Requests Are Sent).
I'd really appreciate if you could help with this. I can share the logs with a DM if needed. Thank you for your time!
