Need advice on handling 10,000 images on Lambda
I need advice, I have a SQS that triggers lambda, it's an image processor that takes in 10,000 images and archives them in a zip file then uploads it to S3.
I'm getting
array buffer allocation
on the archiving part. I might need more memory, currently has a max of 2560mb ram. This might be happening because I'm putting all the archived objects in a buffer instead of streaming, but it doesnt not work if i stream on lambda.
Should I make two lambda functions one for processing the image and uploading everything to s3, and another one for zipping it?2 Replies
@_typedef you can do some stream pipeline stuff with processing, zipping and uploading. It is a while back when I did that on lambda. I'll try to find the code.
nope can't find it anymore. I think was on a client in a private repo.
we do something similar at work where we grab a bunch of PDFs, combine them all to a single zip, and stream them to s3. I think you want to use the s3Client.upload method which takes in a stream for the Body
so you pass the stream from the archiver directly into s3Client.upload({Body: zipStream}) or something