Urgent! all our workers not working! Any network issues?
Please take a look at our workers in endpoint h16kk1hi79s3t0 or kn0n8ry69jj1t7
All the workers are stuck at something!!
42 Replies
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
we're using our custom docker image
how could I create a support ticket?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes, we've running these for months without problem
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes, sure, I'll paste it here
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
two different worker logs. as far as I can see, I think there's definitely some kind of network problems.
These templates have been running for months without any changes.
for the first screenshot, after our logic is done the worker is just not doing anything.
for the second, we do some requests in our docker logic, and it seems these network requests are all failing
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes, all stuck in running state
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I don't know. I'm just guessing there's a network problem in runpod now.
We've been using runpod heaviliy for months and this is quite urgent
These templates have been running without any problem, but since just a few hours ago this problem started happening
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
here's our requests graph.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yeah I can paste the last line, but I don't think this will help you. it's just our docker logic.
2024-05-30T02:45:08.562489094Z exception in main_handler in validation check: <class 'requests.exceptions.ConnectionError'>: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
but please look into it asap.. 🙂
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
we use all the regions. is this what you mean?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
we send a request to amazon s3 to store our image
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes, but we checked locally to send a request to amazon s3, but that works 😦
oh yeah, not only that, we have other things we do.
validation check means.. as far as I remember, we use Amazon Rekognition service to check for nsfw photos
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
we checked that, it works in my computer
the serious thing is, here when it prints "push_output_image" that means our docker logic is done.
normally after that, it should fetch the next runpod job to start, but it's just stuck here
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I think so too.
Would really appreciate it if you could take a look
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
oh no..
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
that would take too long.. I'm just DMing RunPod members when we first started using RunPod a year ago.
Thank you
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
but they're not responding.. are they all off time?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
oh, we're in Korea and I guess it's sleeping time in US..
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
possibly, this is urgent..
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
thanks