Runpod•2y ago•

45 replies

Unacceptably high failed jobs suddenly

Suddenly almost 20% of my serverless jobs failed. I have never had this issue until yesterday. This is is completely UNACCEPTABLE that I am being charged for this immense fuck up and that my customers are being impacted. This needs to be resolved IMMEDIATELY and I demand a refund for this!

ashleyOP•2/26/24, 6:16 AM

@flash-singh @Zeen @JM This is completely UNACCEPTABLE and needs to be RESOLVED IMMEDIATELY and I demand a refund.

ashleyOP•2/26/24, 6:59 AM

Must be some infrastructure issue because I don't even have any error logs for any of my failed jobs.

ashleyOP•2/26/24, 7:00 AM

Also extremely suspicious that the increase in failed jobs coincides with less workers being throttled.

ashleyOP•2/26/24, 7:01 AM

Baran•2/26/24, 8:16 AM

Same here. Not 20% but I still way more than before

ashleyOP•2/26/24, 8:18 AM

Jobs should only fail if there is an error in the severless handler code, which never happened in my case.

ashleyOP•2/26/24, 8:19 AM

I also don't know how this issue is supposed to be debugged when there are no error logs for any of the failed jobs.

ashleyOP•2/26/24, 10:05 AM

Looks like most of them failed due to executionTimeout exceededexecutionTimeout exceeded. My jobs shouldn't take more than 5 minutes to execute, there is something wrong with the workers.

ashleyOP•2/26/24, 10:16 AM

My jobs take 3 minutes max to execute so something is seriously wrong here, and I have been running this endpoint in various different regions for several months and never had this issue until now.

ashleyOP•2/26/24, 10:17 AM

I would also expect to see these executionTimeout errors in the logs for my endpoint, but they aren't in the logs.

ashleyOP•2/26/24, 10:25 AM

Also I don't know why my IP was rate limited on Saturday, this has never happened before and I wasn't even sending that many requests.

ashleyOP•2/26/24, 10:25 AM

This is making serverless more and more unusable by the day.

ashleyOP•2/26/24, 10:32 AM

Each of those 2 terminal windows has a different public IP as well, so there is really no reason why I should have been rate limited.

flash-singh•2/26/24, 4:20 PM

is that 429 error in your handler code or somewhere else?

Fflash-singh is that 429 error in your handler code or somewhere else?

ashleyOP•2/26/24, 4:22 PM

Checking the staus of my jobs mostly.

ashleyOP•2/26/24, 4:22 PM

Also trying to create a new job.

flash-singh•2/26/24, 4:25 PM

why not use webhooks?

ashleyOP•2/26/24, 4:26 PM

Webhooks can be unreliable

flash-singh•2/26/24, 4:27 PM

does this endpoint get a lot of volume?

ashleyOP•2/26/24, 4:31 PM

It fluctuates, weekends are busier than during the week and evenings are busier than the day time (my day time, because most of the customers are in the US).

ashleyOP•2/26/24, 4:32 PM

But there were a lot more requests than usual over the weekend.

ashleyOP•2/26/24, 4:32 PM

Not anything massive though, its on average < 1000 requests per day.

ashleyOP•2/26/24, 4:34 PM

And we havent even done 300 jobs today yet, but it already had 30 failed jobs which is not normal.

ashleyOP•2/26/24, 4:35 PM

C = Completed
F = Failed
R = Retried

C = Completed
F = Failed
R = Retried

ashleyOP•2/26/24, 4:36 PM

And failed graph to display the spike in failed jobs is above.

ashleyOP•2/26/24, 4:36 PM

The table is from the metrics API, and the graph above is from the health API.

ashleyOP•2/26/24, 4:53 PM

Also can't use a webhook because its all running on an internal VPC on AWS which is not publicly accessible.

ashleyOP•2/26/24, 4:55 PM

And if I use a webhook I can't check whether my jobs are stuck on IN_PROGRESS for too long and automatically cancel them.

flash-singh•2/26/24, 6:04 PM

may have to change failed to something else when it times put, cant tell if its that or fail at job level

ashleyOP•2/26/24, 6:07 PM

There is already TIMED_OUTTIMED_OUT can't it just use that?

flash-singh•2/26/24, 6:08 PM

yes thats the plan, in my fix i made it failed, should change it back

ashleyOP•2/26/24, 6:11 PM

Oh yeah, I think its better to change it back

JM•3/1/24, 3:42 AM

Hey @ashleyk

JM•3/1/24, 3:42 AM

Hit me up with your endpoint ID; will provide you credits

JJM Hit me up with your endpoint ID; will provide you credits 👍

octopus•3/1/24, 5:16 AM

Gotta give @ashleyk a job at this point, he helps everyone

JJM Hey @ashleyk

ashleyOP•3/1/24, 6:12 AM

HI @JMendpoint id is sdj01thu7r2mxx. There were issues 24th, 25th, 26th Jan, where my billing escalated more than usual, and the the execution time spiked slightly on 24th and 25th but massively on 26th when there were so many failed jobs. Things seem to have stabilised from 26th Feb onwards.

ashleyOP•3/1/24, 6:13 AM

By the way, there is a large gap because I had to to switch my endpoint to a different region because all my workers were throttled.

JM•3/1/24, 6:20 AM

That's no good, thanks for explaining

JM•3/1/24, 6:21 AM

Btw, I was literally buried in work, I found more hardware for everyone

JM•3/1/24, 6:21 AM

Apologies for delay in responding

JM•3/1/24, 6:23 AM

@ashleyk Credited the account! Thanks for helping everyone

ashleyOP•3/1/24, 6:25 AM

Thats awesome news that you found new hardware @JM , it will make us all very happy, thank you!

. No worries about the delay in responding and thanks very much for the credits, its cool that you can do it directly now and don't need to issue credit codes anymore. Helping people is only a pleasure.

Aashley Thats awesome news that you found new hardware @JM , it will make us all very ha...

JM•3/1/24, 6:26 AM

Yep, engineering has been helping me and Justin very hard lately; new admin features like this one always help so much!

Take care sir, let me know if you need anything. Need to go to bed now

JJM Yep, engineering has been helping me and Justin very hard lately; new admin feat...

ashleyOP•3/1/24, 6:30 AM

Awesome news, thanks very much, you take care too, and have a good nights rest

Unacceptably high failed jobs suddenly

Similar Threads

Similar Threads

Similar Threads