Eren Comments - Answer Overflow

I don't have a1111 ready deployment but i assume it should return that above "error" key in the request fail response, that might be read and then kill worker

26 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

Yeah this applies using your own pipeline mostly wrapping inside of this logic

26 replies

RRunPod

•Created by hammer on 4/8/2025 in #⚡｜serverless

It seems that serverless does not have an option to customize the CPU?

the hardware specs are varying between data-centers on GPU deployments Check your workers tab and see which region it runs, remove that insufficient specs region from Edit>Advanced tab and see what happens. it will assign another worker from another region check it's specs then

9 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

like this:

{
  "delayTime": 2222,
  "error": "bla-bla-bla out of memory i need help to fit 1 mb please bla-bla-bla",
  "executionTime": 2222,
  "id": "223j2kn2b3j23jbk2b",
  "status": "FAILED",
  "workerId": "2j3njkg8fdsmgk"
}

{
  "delayTime": 2222,
  "error": "bla-bla-bla out of memory i need help to fit 1 mb please bla-bla-bla",
  "executionTime": 2222,
  "id": "223j2kn2b3j23jbk2b",
  "status": "FAILED",
  "workerId": "2j3njkg8fdsmgk"
}

26 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

if it polls/checks the request execution result and if it returns FAILED, the error key has the value, it also has this raised "e" variable as string so another way might be that

26 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

many ways but simple as this

try:
    output = model(input_tensor.to("cuda"))
except RuntimeError as e:
    if "out of memory" in str(e).lower():
        print("Caught CUDA OOM – cleaning up")
        torch.cuda.empty_cache()
        gc.collect()
    else:
        raise

try:
    output = model(input_tensor.to("cuda"))
except RuntimeError as e:
    if "out of memory" in str(e).lower():
        print("Caught CUDA OOM – cleaning up")
        torch.cuda.empty_cache()
        gc.collect()
    else:
        raise

26 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

I also strongly recommend implementing torch cache clear support and periodically clearing the garbage collector

26 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

Yes you can catch the OOM exception and use GraphQL API to kill the worker. OOM can occur due to several reasons and that doesn't mean that worker is a poor worker but yeah you can do that programmatically

26 replies

RRunPod

•Created by dbtr on 4/6/2025 in #⚡｜serverless

Serverless endpoint fails with Out Of Memory despite no changes

I can approve that, 1 out of 100 worker having problem to run the code ending up with OOM, just kill the worker and move on

26 replies

RRunPod

•Created by dalton57918 on 3/31/2025 in #⚡｜serverless

Buil docker with environment variables

Your only option is building the docker image, pushing to dockerhub, and pulling from there

12 replies

RRunPod