R
Runpod12mo ago
dhe128

How to get `/stream` serverless endpoint to "stream"?

Example from official documentation: https://docs.runpod.io/sdks/javascript/endpoints#stream
from time import sleep
import runpod


def handler(job):
job_input = job["input"]["prompt"]

for i in job_input:
sleep(1) # sleep for 1 second for effect
yield i


runpod.serverless.start(
{
"handler": handler,
"return_aggregate_stream": True, # Ensures aggregated results are streamed back
}
)
from time import sleep
import runpod


def handler(job):
job_input = job["input"]["prompt"]

for i in job_input:
sleep(1) # sleep for 1 second for effect
yield i


runpod.serverless.start(
{
"handler": handler,
"return_aggregate_stream": True, # Ensures aggregated results are streamed back
}
)
import runpodSdk from "runpod-sdk";

async function main() {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});

console.log(result);

const { id } = result;
for await (const result of endpoint.stream(id)) {
console.log(`${JSON.stringify(result, null, 2)}`);
}
console.log("done streaming");
}
import runpodSdk from "runpod-sdk";

async function main() {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});

console.log(result);

const { id } = result;
for await (const result of endpoint.stream(id)) {
console.log(`${JSON.stringify(result, null, 2)}`);
}
console.log("done streaming");
}
The example suggests that the /stream endpoint should return the entire input, one letter at a time. However, when I try it it only sends the first letter back, followed by the http connection closing. Is /stream intended to be streamed from start to finish, or polled repeatedly?
Endpoints | RunPod Documentation
Learn how to interact with RunPod's endpoints using the JavaScript SDK, including synchronous and asynchronous execution methods, status checks, and job cancellation. Discover how to set timeouts, execute policies, and purge queues.
6 Replies
dhe128
dhe128OP12mo ago
* Looking at js-sdk source: https://github.com/runpod/js-sdk/blob/main/src/index.ts#L203 stream() doesn't pass {responseType: 'stream'} to xior.get so it can't be streaming * Running the runpod UI with network tab open: (https://www.runpod.io/console/serverless/user/endpoint/$endpointId) * It polls /stream every few seconds, not doing streaming.
Unknown User
Unknown User12mo ago
Message Not Public
Sign In & Join Server To View
rinarakaki
rinarakaki12mo ago
Is it true only for the js client or the serverless runtime entirely?
it doesn't actually "stream"
If so, it really must be documented because to all of us 'stream' means 'chunked transfer encoding', not the general category of usage encompassing polling. I was suspicious why they don't have -N/--no-buffer flag in their curl example for the stream endpoint but now have figured out why.. https://docs.runpod.io/serverless/endpoints/job-operations#stream-results
Job operations | RunPod Documentation
Learn how to use the Runpod Endpoint to manage job operations, including running, checking status, purging queues, and streaming results, with cURL and SDK examples.
Unknown User
Unknown User12mo ago
Message Not Public
Sign In & Join Server To View
dhe128
dhe128OP12mo ago
thank you! I got it working with stream via polling :scarlettHeart:
Unknown User
Unknown User12mo ago
Message Not Public
Sign In & Join Server To View

Did you find this page helpful?