5 replies

Need advice on handling 10,000 images on Lambda

I need advice, I have a SQS that triggers lambda, it's an image processor that takes in 10,000 images and archives them in a zip file then uploads it to S3.

I'm getting

array buffer allocation

array buffer allocation

on the archiving part. I might need more memory, currently has a max of 2560mb ram. This might be happening because I'm putting all the archived objects in a buffer instead of streaming, but it doesnt not work if i stream on lambda.

archive.finalize();
const buffer = [];
archive.on("data", (data => buffer.push(data)));

archive.on('end', (async () => {
  const data = Buffer.concat(buffer);

  await putObjectToS3(`generator/${userId}/collection.zip`, data);
  console.info(`<${userId}>`, "sent zip file to s3");

  await collection.updateOne({ userId }, { $set: { status: "completed" } });

  resolve(formatResponse({
    statusCode: 200,
    body: { code: "SUCCESS" },
  }))
}))

archive.finalize();
const buffer = [];
archive.on("data", (data => buffer.push(data)));

archive.on('end', (async () => {
  const data = Buffer.concat(buffer);

  await putObjectToS3(`generator/${userId}/collection.zip`, data);
  console.info(`<${userId}>`, "sent zip file to s3");

  await collection.updateOne({ userId }, { $set: { status: "completed" } });

  resolve(formatResponse({
    statusCode: 200,
    body: { code: "SUCCESS" },
  }))
}))

Should I make two lambda functions one for processing the image and uploading everything to s3, and another one for zipping it?

TTC

Theo's Typesafe Cult•3y ago•

5 replies

typedef

Need advice on handling 10,000 images on Lambda

I need advice, I have a SQS that triggers lambda, it's an image processor that takes in 10,000 images and archives them in a zip file then uploads it to S3.

I'm getting

array buffer allocation

array buffer allocation

archive.finalize();
const buffer = [];
archive.on("data", (data => buffer.push(data)));

archive.on('end', (async () => {
  const data = Buffer.concat(buffer);

  await putObjectToS3(`generator/${userId}/collection.zip`, data);
  console.info(`<${userId}>`, "sent zip file to s3");

  await collection.updateOne({ userId }, { $set: { status: "completed" } });

  resolve(formatResponse({
    statusCode: 200,
    body: { code: "SUCCESS" },
  }))
}))

archive.finalize();
const buffer = [];
archive.on("data", (data => buffer.push(data)));

archive.on('end', (async () => {
  const data = Buffer.concat(buffer);

  await putObjectToS3(`generator/${userId}/collection.zip`, data);
  console.info(`<${userId}>`, "sent zip file to s3");

  await collection.updateOne({ userId }, { $set: { status: "completed" } });

  resolve(formatResponse({
    statusCode: 200,
    body: { code: "SUCCESS" },
  }))
}))

Should I make two lambda functions one for processing the image and uploading everything to s3, and another one for zipping it?

Need advice on handling 10,000 images on Lambda

Similar Threads

Need advice on handling 10,000 images on Lambda

Similar Threads

Similar Threads

Similar Threads