Runpod•5mo ago

RunPod Serverless Endpoint Issue - Jobs Complete But No Output Returned

Problem: My serverless endpoint jobs are completing successfully but returning empty results.

Endpoint ID: su6ufhaephnw03 (Stable Diffusion XL)

Symptoms:

Jobs submit successfully (200 OK)
Status shows COMPLETED
Stream endpoint returns {"status": "COMPLETED", "stream": []}
No output/images returned despite successful completion
Example Job:

Job ID: 3906a491-6431-48f2-8c6b-afe60e5f5be0-u1
Status: COMPLETED
Stream: Empty array
What I've Tried:

Simplified payload to minimal parameters

Fixed tensor precision issues (use_fp16: false)

Verified API authentication and endpoints

Multiple test runs with same result
Timeline: Issue started around July 31st, affecting all jobs since then.

Question: Is there a known issue with result retrieval for this endpoint? The jobs appear to process (they take normal execution time ~1s) but no output is returned through either the status or stream endpoints.

Additional Info: Using Python requests library, following standard serverless API patterns. Happy to provide more technical details if needed.

Any guidance would be appreciated!

Jason•8/1/25, 1:44 AM

try logging what you will return in the worker, so you could check worker logs, and log all errors inside the worker

bizetoolsOP•8/1/25, 1:47 AM

Hey Jason!

Thanks for the suggestion to check worker logs. Here's what I found:

GOOD NEWS: My precision fix worked!

Recent jobs show no more tensor errors
Pipeline loads successfully (100% complete)
Jobs go from Started → Finished cleanly
MYSTERY: No handler return logging

2025-08-01 11:06:35.765 | info | 9x9qgwlpdx4b4j | Started.
2025-08-01 11:06:35.765 | info | 9x9qgwlpdx4b4j | Finished.
Missing from logs:

No return value logging
No handler output statements
No errors between Started/Finished
Question: Should the handler be logging what it's trying to return? The worker completes but I'm getting empty streams: {"status": "COMPLETED", "stream": []}

My payload (minimal):

json
{
"input": {
"prompt": "A realistic photograph of a modern office building",
"use_fp16": false,
"precision": "fp32"
}
}
Is there a way to add debug logging to see what the handler is actually returning before it hits the result callback system?

Bbizetools Hey Jason! 👋 Thanks for the suggestion to check worker logs. Here's what I fou...

Jason•8/1/25, 1:51 AM

ohh thanks for the detailed update, yes try to log it to be sure if anything's wrong

Jason•8/1/25, 1:52 AM

oryou can copy the whole handler file link (on github) or codes that is relevant here (but if you cut out certain pieces it will be hard to figure it out)

bizetoolsOP•8/1/25, 1:53 AM

K thanks !! Really appreciated

Jason•8/1/25, 1:53 AM

did your worker worked before (returns normally) but suddenly becomes like this without you changing the container image?

bizetoolsOP•8/1/25, 1:55 AM

correct. Worked previously then no more

bizetoolsOP•8/1/25, 1:55 AM

New Endpoint:

Creating fresh endpoint with 24GB GPU config - should be ready shortly.

Handler Code Request: I don't have direct access to the handler file in the RunPod container, but I suspect it's the standard Stable Diffusion XL template from RunPod's marketplace.

Key Questions:

Should the handler be logging return values? Currently seeing Started → Finished with no output logs
Is there a standard way to add debug logging to see what's being returned before the callback?
Could this be a template issue? The handler might be returning data in a format the callback system doesn't recognize
Current Behavior:

Jobs complete successfully

No errors in recent runs

Stream returns empty: {"status": "COMPLETED", "stream": []}

No handler return logging visible
Next Steps:

Testing new endpoint once created
Happy to share any specific handler code if you can point me to where it's accessible
Would love guidance on adding debug logging to troubleshoot return values
Should I be looking somewhere specific for the handler file, or is this typically built into the RunPod template?

Jason•8/1/25, 1:55 AM

i see

Jason•8/1/25, 1:56 AM

what you stream in your handler

Jason•8/1/25, 1:56 AM

Should the handler be logging return values? Currently seeing Started → Finished with no output logs ye thats what i suggested just now

Is there a standard way to add debug logging to see what's being returned before the callback? print is fine but you can use runpod's logging (check runpod-python on github)

Could this be a template issue? The handler might be returning data in a format the callback system doesn't recognize trying to figure this out rn

bizetoolsOP•8/1/25, 1:58 AM

Will do, thanks again

Jason•8/1/25, 2:03 AM

maybe send your handler file if you can

bizetoolsOP•8/1/25, 2:15 AM

Handler File Location: The handler file is inside the RunPod container, not in my local codebase. I'm using the standard Stable Diffusion XL template from RunPod's marketplace.

What I have access to:

My client-side batch processor code

API requests/responses

Worker logs from RunPod

The actual handler file inside the container
Questions:

Where can I find the handler file? Is it accessible through RunPod's interface or GitHub?
Is this the standard SDXL template handler? If so, do you have a link to the source?
Can I customize/override the handler? To add debug logging for return values?
Current Evidence:

Jobs complete successfully (Started → Finished)
No handler return logging visible in worker logs
Stream returns empty: {"status": "COMPLETED", "stream": []}
Using minimal payload: {"prompt": "...", "use_fp16": false, "precision": "fp32"}
My batch processor code (client-side) is working perfectly - the issue seems to be in the container's handler not returning data properly.

Would love guidance on accessing/debugging the actual handler!

bizetoolsOP•8/1/25, 2:23 AM

Created new endpoint
Old endpoint pattern:

Started. → Finished. → Failed to return job results.
New endpoint pattern:

Started. → Failed to return job results. (NO "Finished" log!)
This suggests:

Old endpoint: Job completes but can't return results
New endpoint: Job fails during execution (no "Finished" log)
Both endpoints have the same result callback failure (400 Bad Request), but the new endpoint isn't even finishing the job.

Question: Should there always be a Finished log when jobs complete? The missing Finished log on the new endpoint suggests the handler is crashing or failing silently during image generation.

Job ID for debugging: 0c67a99b-cfe1-4b5d-bd6e-a03da9bae9b7-u1 (worker: 3fhwe3blgxcl2h)

This pattern difference might be the key to understanding what's happening

Jason•8/1/25, 2:23 AM

whats your return payload lie?

Jason•8/1/25, 2:23 AM

what do you return?

Jason•8/1/25, 2:23 AM

+How big is it?

8/1/25, 2:24 AM

Jason•8/1/25, 2:24 AM

okay maybe just go talk with support :)

Jason•8/1/25, 2:24 AM

its hard for me to see whats happening this way

bizetoolsOP•8/1/25, 2:25 AM

You are a great help. Thanks heaps!

beweinreich•9/9/25, 2:55 PM

@bizetools @Jason did you figure out what happened here? I am having the same issue.

RunPod Serverless Endpoint Issue - Jobs Complete But No Output Returned

Similar Threads

Similar Threads

Similar Threads