R
Runpod6mo ago
light

Stuck when run is triggered via API call but not on dashboard?

I have a project that let's me upload videos on google cloud storage (it is very bare and that's the only thing that it does at the moment). If I trigger the request form serverless dashboard, the job gets completed, but if it is triggered via API it is stuck forever this is what the code looks like:
const requestConfig = {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${runpod.api_key}`,
},
body: JSON.stringify({
input: {
video_path,
audio_path,
},
}),
};

const response = await fetch(https://api.runpod.ai/v2/*****/run`, requestConfig);
const requestConfig = {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${runpod.api_key}`,
},
body: JSON.stringify({
input: {
video_path,
audio_path,
},
}),
};

const response = await fetch(https://api.runpod.ai/v2/*****/run`, requestConfig);
Any Ideas?
No description
No description
44 Replies
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
@Jason not 100% sure but I have a hunch that it gets stuck in this function (based on the progress message)
def generate_signed_url(blob_name, expiration_minutes=60):
log.info(f"Generating signed URL for {blob_name}")
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(expiration=timedelta(minutes=expiration_minutes))
log.info(f"Signed URL: {url}")
return url
def generate_signed_url(blob_name, expiration_minutes=60):
log.info(f"Generating signed URL for {blob_name}")
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(expiration=timedelta(minutes=expiration_minutes))
log.info(f"Signed URL: {url}")
return url
but that doesn't explain why it works if I trigger it via dashboard but not via API?
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
sorry, this is what my handler looks like - it doesn't really do anything at the moment other than downloading the same file and uploading it to GCS
light
lightOP6mo ago
and this is what my docker looks like:
# Use Runpod base image
FROM runpod/base:0.4.0-cuda11.8.0

# Optional: Install extra Linux packages here if needed
# COPY builder/setup.sh /setup.sh
# RUN bash /setup.sh && rm /setup.sh

# Set working directory
WORKDIR /app

# Copy Python dependencies
COPY builder/requirements.txt .

# Install Python dependencies
RUN python3.11 -m pip install --upgrade pip && \
python3.11 -m pip install --upgrade -r requirements.txt --no-cache-dir && \
rm requirements.txt

# Copy your actual src code
COPY src/ .

# Start the worker
CMD ["python3.11", "-u", "handler.py"]
# Use Runpod base image
FROM runpod/base:0.4.0-cuda11.8.0

# Optional: Install extra Linux packages here if needed
# COPY builder/setup.sh /setup.sh
# RUN bash /setup.sh && rm /setup.sh

# Set working directory
WORKDIR /app

# Copy Python dependencies
COPY builder/requirements.txt .

# Install Python dependencies
RUN python3.11 -m pip install --upgrade pip && \
python3.11 -m pip install --upgrade -r requirements.txt --no-cache-dir && \
rm requirements.txt

# Copy your actual src code
COPY src/ .

# Start the worker
CMD ["python3.11", "-u", "handler.py"]
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
I cancel it when it gets too long, but sometimes I come back after around 30mins or so and it is still processing, but when I trigger it via dashboard it would only take a couple of seconds
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
I mean "In Progress" status
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
or there might be some error? that keeps retrying
the code I sent you was a refactored version, this was the first version that has the same issue, at first I thought something is failing that's why it stuck so i added retry event and error handlers, but that doesn't seem to be the case
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
so this one, I tried running this like 3 minutes ago and still processing. it is stuck in Generating signed URL..
No description
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
this is what the logs look like
No description
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
the RunPodLogger doesn't seem to work
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
riverfog7
riverfog76mo ago
is the handler running? why does it stop immediately
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
all the workers are on idle as well lemme try using print and update you guys.
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
I have adjusted the concurrency to return 5 and use print() directly but it is still the same 🤔
No description
light
lightOP6mo ago
I even waited for the build to complete sorry, how would I know if the handler is running?
light
lightOP6mo ago
HI @Jason ok, it looks like the logs are transitioned to the one above once it is "completed" and I was able to capture the worker logs before it transitioned, as you can see on this log it says "Job completed successfully" there was even a "Finished." message after that, and this is what my code looks like (file attached). the
print("✅ Job completed successfully.")
print("✅ Job completed successfully.")
is right before my return on my handler, yet the status of my job is still tagged as IN PROGRESS
Poddy
Poddy6mo ago
@light
Escalated To Zendesk
The thread has been escalated to Zendesk!
Ticket ID: #16867
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
Hmm this is kind of weird try to open a ticket
Thank you so much!
riverfog7
riverfog76mo ago
What happens if you do runsync? instead of run
light
lightOP6mo ago
@riverfog7 ok I just tried it, very weird thing is happening: If I run this from my proxy API (this is in NodeJS):
const requestConfig = {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${runpod.api_key}`,
},
body: JSON.stringify({
input: {
video_path,
audio_path,
},
}),
};

const response = await fetch(`${runpod.endpoint}/runsync`, requestConfig);
const data = await response.json();

return data;
const requestConfig = {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${runpod.api_key}`,
},
body: JSON.stringify({
input: {
video_path,
audio_path,
},
}),
};

const response = await fetch(`${runpod.endpoint}/runsync`, requestConfig);
const data = await response.json();

return data;
data would return:
{
"delayTime": 5150,
"executionTime": 2775,
"id": <job id>,
"output": {
"message": "Files processed, uploaded, and cleaned up successfully",
"signed_url": "..."
},
"status": "COMPLETED",
"workerId": "83kgwesbus3cfb"
}
{
"delayTime": 5150,
"executionTime": 2775,
"id": <job id>,
"output": {
"message": "Files processed, uploaded, and cleaned up successfully",
"signed_url": "..."
},
"status": "COMPLETED",
"workerId": "83kgwesbus3cfb"
}
meaning it has been COMPLETED BUT If I visit the dashboard it would show as IN PROGRESS, calling the API to check for the status of job id, it would return IN PROGRESS then after a couple of minutes it would get removed from my "requests" panel leaving no trace of that ever happening. Although In reality, I will not use runsync in my project since it is a proxy API and I do not want to have timeout errors if my process in the future would take too long to finish. I would rather use run and poll the status.
riverfog7
riverfog76mo ago
the request getting removed part is expected i think i saw that runsync operations have short TTL in the docs anyways the API returning IN_PROGRESS is strange and must be a runpod side problem
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
yes that is correct, I will try to record my screen later to show what it looks like.
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
I clicked the "Open ZenDesk ticket" above, are there any way of opening a ticket or that's it? (I may have entered the wrong email, but I guess that's fine?)
So on /runsync it says it's done but in status it's not?
exactly.
riverfog7
riverfog76mo ago
It looks like a product demo😆 😆 Anyways its 100% runpod's fault
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
light
lightOP6mo ago
haha kind of an off topic but - we are using AI features to make screen recording look more professional with few edits (mine is very bare tho as I did not even edit it)
Unknown User
Unknown User6mo ago
Message Not Public
Sign In & Join Server To View
₳Ñ₳𝕄îӃ₳
@light : I have updated your ticket with some suggestion, can you please try that when you get a chance
light
lightOP6mo ago
Hi @₳Ñ₳𝕄îӃ₳ I replied on the email, but will copy-paste my reply here:
It is still bugged, but now it is stuck in progress even if I trigger it directly from the dashboard:

here is a recording.
https://app.canvid.com/share/fi_01JTFC00JSQMJPP236V39N3NKW

As you can see, in the code and even on the "progress output" it is not stuck on the "Done" but the status is in progress, however it seems that jobs triggered via API is now getting completed..

This is very weird
It is still bugged, but now it is stuck in progress even if I trigger it directly from the dashboard:

here is a recording.
https://app.canvid.com/share/fi_01JTFC00JSQMJPP236V39N3NKW

As you can see, in the code and even on the "progress output" it is not stuck on the "Done" but the status is in progress, however it seems that jobs triggered via API is now getting completed..

This is very weird
₳Ñ₳𝕄îӃ₳
@light : Can you please have a look in the ticket when you get a chance. I need few details to involve our engineering team.
light
lightOP6mo ago
@₳Ñ₳𝕄îӃ₳ sure, I have sent you 2 recordings of what is happenning and explaining the code. Let me know if it helps.

Did you find this page helpful?