Worker logs say this to all requests: API: unreachable.. retrying in 100ms
Logs of one of my workers say "API: unreachable.. retrying in 100ms" to all requests. See screenshot attached.
Then Runpod's API replies the following:
{ "delayTime": 10776,
"executionTime": 10206,
"id": "03fec117-c135-435b-bed1-577883d280d9-e1",
"output": "Error queuing workflow: <urlopen error [Errno 111] Connection refused>",
"status": "COMPLETED"}
The last item in the logs (other than "API: unreachable.. retrying in 100ms") says:
S[runpod-worker-comfy] WebSocket connection closed: no close frame received or sent
After this log the worker returned only "API: unreachable.. retrying in 100ms" and response as seen above.
Any ideas what may have caused this behavior and how to avoid it?
Other workers with the same docker image work fine.
7 Replies
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
That's sad, but sounds right 😦
Do you think there is a way to handle this error? For example, force restart the worker somehow?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yeah - it's a custom image.
Manually setting max workers to 0 and then back works.
Maybe there is a way to reload workers programmatically?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
Yes - if I set max workers to 0 and then back to some number - the problem goes away for some time.
Will investigate further.
Thanks for the advise about graphql. At least I can reload all workers programmatically.
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View