CD
Cloudflare Developers•4d ago
aldenks

How to stream responses and return a known content length?

I'm using workers as a thin proxy with a little bit of logic between object storage (s3 in this case, but in the future likely r2) and clients. I'm wondering how to support sending data back with a content length header but without buffering the full object in memory? When I return a response with a stream, workers transforms the response to Transfer-Encoding: chunked and removes the Content-Length header to conform to the http spec. There's a great http client (object_store) I'd like to support with this proxy, but it requires content length headers. When i asked there about transfer encoding chunked support they said nope (https://github.com/apache/arrow-rs-object-store/issues/340#issuecomment-2849485808). Any suggestions?
GitHub
Support Transfer-Encoding: chunked responses in HttpStore · Issu...
Is your feature request related to a problem or challenge? Please describe what you are trying to do. I'm not able to use the HttpStore to read data returned with Transfer-Encoding: chunked. In...
1 Reply
aldenks
aldenksOP•4d ago
An example of the relevant code. I am seeing the X-Content-Length header so I know it's hitting that branch but the Content-Length header is removed (following spec). I do want to follow http spec. I'm wondering if there's a way for workers to send the data along without be buffering in memory and with a content length header? HTTP2+ not having transfer encoding chunked at all seems to suggest there might be a way? 
const s3Response = await getS3Object(catalogItem.s3Region, fullS3Url, rangeHeader);

const headers = new Headers({
    'Content-Type': 'application/octet-stream',
});

setCacheHeaders(headers, path);

if (s3Response.ContentLength != null) {
    headers.set('Content-Length', s3Response.ContentLength.toString());
    headers.set('X-Content-Length', s3Response.ContentLength.toString());
}

const response = new Response(s3Response.Body as ReadableStream, {
    headers,
    status: 200,
});
const s3Response = await getS3Object(catalogItem.s3Region, fullS3Url, rangeHeader);

const headers = new Headers({
    'Content-Type': 'application/octet-stream',
});

setCacheHeaders(headers, path);

if (s3Response.ContentLength != null) {
    headers.set('Content-Length', s3Response.ContentLength.toString());
    headers.set('X-Content-Length', s3Response.ContentLength.toString());
}

const response = new Response(s3Response.Body as ReadableStream, {
    headers,
    status: 200,
});

Did you find this page helpful?