Theo's Typesafe Cult•12mo ago

Using JSZip with Zod and tRPC

I am making a uploadRouter for my website where a user can upload 5-10 images. I want to be able to zip those image and then upload it to a r2 endpoint. I never really worked with zipping and image schemas in Zod and tRPC so im not sure on what to do. Here is what I have so far

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const imageSchema = z.object({
    image: z
        .any()
        .refine((file) => file?.size <= MAX_FILE_SIZE, 'Maximum file size is 50MB')
        .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file?.type), 'Unsupported file type')
})

export const uploadRouter = router({
    upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))
        .mutation(({ ctx, input }) => {

            const zip = new JSZip();
            const remoteZips = input.map((image) => {
                // ERROR HERE
                zip.file(image.name, image)
            });
            
        })
})

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const imageSchema = z.object({
    image: z
        .any()
        .refine((file) => file?.size <= MAX_FILE_SIZE, 'Maximum file size is 50MB')
        .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file?.type), 'Unsupported file type')
})

export const uploadRouter = router({
    upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))
        .mutation(({ ctx, input }) => {

            const zip = new JSZip();
            const remoteZips = input.map((image) => {
                // ERROR HERE
                zip.file(image.name, image)
            });
            
        })
})

This is the error im getting

No overload matches this call.
  Overload 1 of 4, '(path: string, data: null, options?: (JSZipFileOptions & { dir: true; }) | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'null'.
  Overload 2 of 4, '(path: string, data: string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream>, options?: JSZipFileOptions | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] |

No overload matches this call.
  Overload 1 of 4, '(path: string, data: null, options?: (JSZipFileOptions & { dir: true; }) | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'null'.
  Overload 2 of 4, '(path: string, data: string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream>, options?: JSZipFileOptions | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] |

59 Replies

Grey•12mo ago

const zip = new JSZip();

input.forEach((file) => {
        const data = fs.readFileSync(file.path);
        zip.file(file.originalname, data);
      });

const content = await zip.generateAsync({ type: 'nodebuffer' });
return content;

const zip = new JSZip();

input.forEach((file) => {
        const data = fs.readFileSync(file.path);
        zip.file(file.originalname, data);
      });

const content = await zip.generateAsync({ type: 'nodebuffer' });
return content;

try this you're using it wrong

elpupper•12mo ago

hmm my schema is most def wrong

Grey•12mo ago

also, maybe this should be a better zod schema

z.array(z.object({
  fieldname: z.string().refine((value) => value === 'images', {
    message: 'Invalid fieldname',
  }),
  originalname: z.string().min(3),
  encoding: z.string().optional(),
  mimetype: z.string().refine((value) => value.startsWith('image/'), {
    message: 'Invalid file type',
  }),
  destination: z.string().optional(),
  filename: z.string().optional(),
  path: z.string().optional(),
  size: z.number().positive(),
}));

z.array(z.object({
  fieldname: z.string().refine((value) => value === 'images', {
    message: 'Invalid fieldname',
  }),
  originalname: z.string().min(3),
  encoding: z.string().optional(),
  mimetype: z.string().refine((value) => value.startsWith('image/'), {
    message: 'Invalid file type',
  }),
  destination: z.string().optional(),
  filename: z.string().optional(),
  path: z.string().optional(),
  size: z.number().positive(),
}));

elpupper•12mo ago

yep

Grey•12mo ago

you're using 1 object it should be an array

elpupper•12mo ago

upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))

upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))

Grey•12mo ago

hmm didn't saw that

elpupper•12mo ago

its not making any sense to me what u sent nor does mine either cause it was just ripped from a blog and even the blog didnt make sense basically what i need is to get the images that the user uploaded so for example i select 5 to 10 images on the website click upload then it goes to this trpc endpoint and zips them

Grey•12mo ago

ya know....... you can just.... use a nextjs raw http endpoint to handle file uploads

elpupper•12mo ago

hmm that is true

Grey•12mo ago

I've always used those for doing file stuff as the raw request object is available

elpupper•12mo ago

ok im gonna do that is there a guide or documentation u can link me to?

Grey•12mo ago

hold up I have a code snippet I wrote a few days ago hmm this is what I wrote, it's saving one image a time though you'll have to read the formidable docs to handle first parsing multiple files, then saving them. Or hold up a minute wait don't use that it'll get messier that way

elpupper•12mo ago

alr i just want a way to get the images and check if they are actually images and check if they are smaller than 50 mb thats it

Grey•12mo ago

I have something along those lines, let me remember where I saw it *proompting some code into existance here

import { NextApiRequest, NextApiResponse } from 'next';
import multer from 'multer';
import JSZip from 'jszip';
import fs from 'fs';

const upload = multer({ dest: 'uploads/' });

// Disable built-in body parsing
export const config = {
  api: { bodyParser: false },
};

export default upload.array('images', 5, async (req: NextApiRequest, res: NextApiResponse) => {
  try {
    const zip = new JSZip();

    req.files.forEach((file: Express.Multer.File) => {
      const data = fs.readFileSync(file.path);
      zip.file(file.originalname, data);
    });

    const zipContent = await zip.generateAsync({ type: 'nodebuffer' });

    // Save the zip file
    const zipFilePath = 'path/to/save/zip.zip';
    fs.writeFileSync(zipFilePath, zipContent);

    res.status(200).json({ message: 'Zip file saved successfully' });
  } catch (error) {
    console.error('Error saving zip file:', error);
    res.status(500).json({ message: 'Failed to save zip file' });
  } finally {
    // Cleanup the uploaded files
    req.files.forEach((file: Express.Multer.File) => {
      fs.unlinkSync(file.path);
    });
  }
});

import { NextApiRequest, NextApiResponse } from 'next';
import multer from 'multer';
import JSZip from 'jszip';
import fs from 'fs';

const upload = multer({ dest: 'uploads/' });

// Disable built-in body parsing
export const config = {
  api: { bodyParser: false },
};

export default upload.array('images', 5, async (req: NextApiRequest, res: NextApiResponse) => {
  try {
    const zip = new JSZip();

    req.files.forEach((file: Express.Multer.File) => {
      const data = fs.readFileSync(file.path);
      zip.file(file.originalname, data);
    });

    const zipContent = await zip.generateAsync({ type: 'nodebuffer' });

    // Save the zip file
    const zipFilePath = 'path/to/save/zip.zip';
    fs.writeFileSync(zipFilePath, zipContent);

    res.status(200).json({ message: 'Zip file saved successfully' });
  } catch (error) {
    console.error('Error saving zip file:', error);
    res.status(500).json({ message: 'Failed to save zip file' });
  } finally {
    // Cleanup the uploaded files
    req.files.forEach((file: Express.Multer.File) => {
      fs.unlinkSync(file.path);
    });
  }
});

here an example of using multer to handle files I used this once before so I remember it works cleanly with files. mind you the snippet above is from ChatGPT so use at your own discretion

elpupper•12mo ago

are u using chatgpt lol

Grey•12mo ago

haha just did as an example, I don't have a code snippet for multer

elpupper•12mo ago

whats multer?

Grey•12mo ago

https://github.com/expressjs/multer#readme

GitHub

GitHub - expressjs/multer: Node.js middleware for handling `multipa...

Node.js middleware for handling multipart/form-data. - GitHub - expressjs/multer: Node.js middleware for handling multipart/form-data.

elpupper•12mo ago

also i want this to be protected thats why trpc made sense

Grey•12mo ago

put in some pre-checks like in the above code snippet, before the try block, put in your

const session = await redis.get(req.cookies.get("SID")); 
if (!isSessionValid(session)) {
  return Response("Invalid Request", {status: 401})
}

const session = await redis.get(req.cookies.get("SID")); 
if (!isSessionValid(session)) {
  return Response("Invalid Request", {status: 401})
}

or whatever session validation you have

elpupper•12mo ago

quick update im fully committed to using trpc and to making it work so currently i have

// create.tsx
const [images, setImages] = useState<File[]>([]);

// create.tsx
const [images, setImages] = useState<File[]>([]);

Grey•12mo ago

👍 There's a way, good luck

elpupper•12mo ago

i just did it 😎

Grey•12mo ago

GGs, did you inject the http req body into the context? or did you do it in another way?

elpupper•12mo ago

first i made my own type

type Image = {
    name: string;
    type: string;
    size: number;
    base64: string;
}

type Image = {
    name: string;
    type: string;
    size: number;
    base64: string;
}

Grey•12mo ago

mmhm

elpupper•12mo ago

then made a function to get the base64 (client side)

    const getBase64 = (file: File) => {
        return new Promise((resolve) => {
            const reader = new FileReader();
            reader.readAsDataURL(file);
            reader.onload = () => resolve(reader.result);
        });
    }

    const getBase64 = (file: File) => {
        return new Promise((resolve) => {
            const reader = new FileReader();
            reader.readAsDataURL(file);
            reader.onload = () => resolve(reader.result);
        });
    }

Grey•12mo ago

okay and

elpupper•12mo ago

then in the file input i got

                                    <input
                                        id="dropzone-file"
                                        type="file"
                                        multiple
                                        onChange={(e) => {
                                            // we need to convert the files to base64
                                            const files = e.target.files;
                                            if (files) {
                                                const filesArray = Array.from(files);
                                                const promises = filesArray.map(async (file) => {
                                                    const base64 = await getBase64(file);
                                                    return {
                                                        name: file.name,
                                                        type: file.type,
                                                        size: file.size,
                                                        base64: base64!.toString(),
                                                    };
                                                });

                                                Promise.all(promises).then((images_) => {
                                                    setImages((prevImages) => images_);
                                                });
                                            }
                                        }}
                                        accept="image/png, image/jpeg, image/jpg"
                                        className="hidden" />

                                    <input
                                        id="dropzone-file"
                                        type="file"
                                        multiple
                                        onChange={(e) => {
                                            // we need to convert the files to base64
                                            const files = e.target.files;
                                            if (files) {
                                                const filesArray = Array.from(files);
                                                const promises = filesArray.map(async (file) => {
                                                    const base64 = await getBase64(file);
                                                    return {
                                                        name: file.name,
                                                        type: file.type,
                                                        size: file.size,
                                                        base64: base64!.toString(),
                                                    };
                                                });

                                                Promise.all(promises).then((images_) => {
                                                    setImages((prevImages) => images_);
                                                });
                                            }
                                        }}
                                        accept="image/png, image/jpeg, image/jpg"
                                        className="hidden" />

pretty much we just convert it to image type

Grey•12mo ago

I see

elpupper•12mo ago

then the upload.ts router is almost the same

Grey•12mo ago

so you're abusing a base64 string of an image to uploading it normally to trpc

elpupper•12mo ago

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";
import { promises as fsPromises } from 'fs';

const { writeFile } = fsPromises;

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const uploadSchema = z.object({
    imageFiles: z
        .array(z
            .any()
            .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file.type), {
                message: "Invalid file type",
            })
            .refine((file) => file.size <= MAX_FILE_SIZE, {
                message: "File is too large",
            })
        )
        .min(5)
        .max(10),
});

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";
import { promises as fsPromises } from 'fs';

const { writeFile } = fsPromises;

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const uploadSchema = z.object({
    imageFiles: z
        .array(z
            .any()
            .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file.type), {
                message: "Invalid file type",
            })
            .refine((file) => file.size <= MAX_FILE_SIZE, {
                message: "File is too large",
            })
        )
        .min(5)
        .max(10),
});

export const uploadRouter = router({
    create: protectedProcedure
        .input(uploadSchema)
        .mutation(async ({ ctx, input }) => {
            console.log("Creating zip file");
            const { imageFiles } = input;
            const zip = new JSZip();

            console.log("Adding files to zip");
            imageFiles.forEach((file, index) => {
                console.log(`Adding file ${index + 1} of ${imageFiles.length}`);
                console.log(`File name: ${file.name}`);
                file.base64 = file.base64.replace(/^data:image\/(png|jpeg|jpg|webp);base64,/, "");
                file.data = Buffer.from(file.base64, "base64");
                zip.file(file.name, file.data);
            });

            const zipFile = await zip.generateAsync({ type: "nodebuffer" });

            console.log("Saving zip file");

            await writeFile("images.zip", zipFile);

            return { success: true };
        })
})

export const uploadRouter = router({
    create: protectedProcedure
        .input(uploadSchema)
        .mutation(async ({ ctx, input }) => {
            console.log("Creating zip file");
            const { imageFiles } = input;
            const zip = new JSZip();

            console.log("Adding files to zip");
            imageFiles.forEach((file, index) => {
                console.log(`Adding file ${index + 1} of ${imageFiles.length}`);
                console.log(`File name: ${file.name}`);
                file.base64 = file.base64.replace(/^data:image\/(png|jpeg|jpg|webp);base64,/, "");
                file.data = Buffer.from(file.base64, "base64");
                zip.file(file.name, file.data);
            });

            const zipFile = await zip.generateAsync({ type: "nodebuffer" });

            console.log("Saving zip file");

            await writeFile("images.zip", zipFile);

            return { success: true };
        })
})

there it is

Grey•12mo ago

nice workaround haha, I might have thought of that on my 2nd-3rd try if I was as adamant as you in making trpc work with it

elpupper•12mo ago

thats all the code i also had to disable the 1mb limit u should probably increase it someone did warn me that there was a 5mb limit in prod so i will try that now

Grey•12mo ago

which is not a good thing to do generally (payload bombing to slow the server down/^ cost)

elpupper•12mo ago

hmm

Grey•12mo ago

as long as you keep a limit, it's probably alright

elpupper•12mo ago

but this is with images so i think its fine

Grey•12mo ago

there is already WAFs on serverless

elpupper•12mo ago

i have the limit set in zod to 50mb per image

Grey•12mo ago

it is

elpupper•12mo ago

so it is fine

Grey•12mo ago

hmm but..... are your users going to be uploading 8k raw images? even those don't touch 50mb

elpupper•12mo ago

nah your right

Grey•12mo ago

you should reduce it to around 10mb per image

elpupper•12mo ago

i should prolly leave it around 10mb yep

Grey•12mo ago

👋 well may not be the best solution, but definitely a good one ( I did that too the other day with a custom captcha service I made in go )

elpupper•12mo ago

yep i hope there is a better solution out there with trpc but this is what i came up with and i couldnt find any online resources

Grey•12mo ago

there is, it's just that it's focused on web standard file uploading, where as you're seeing it with the eye of t3 stack's abstracted layer; which I'm sure does not have any references online which is why I recommended a raw next http endpoint in the first place

elpupper•12mo ago

in the end i think it would be the same

Grey•12mo ago

it actually might be a bit slower

elpupper•12mo ago

how so

Grey•12mo ago

but yeah if it's serverless, you're not measuring the server resource costs costs = CPU/RAM usage, which is negligible for a small userbase, but noticable on the 10k+ user base

elpupper•12mo ago

awe whatever our product has like a 10000% margin

Grey•12mo ago

eh trpc setting up it's context to process and all, plus you using b64 to encode and decode 5 images, the cpu does it's work encoding/decoding comes with a cost, just that it's not noticable on the smaller scale. The reason for there being url-form-encoded is to optimize image transfers over http image= file transfers anyhow that's me being over-conscious of the costs 👋 well whatever fits your need is a good solution

elpupper•12mo ago

hope someone stumbles upon this and finds it useful and if they have a better solution they should tag me

Grey•12mo ago

maybe I'll do it IF I ever decide to handle file uploads in this interesting manner I know the better solution...... just that I'm a bit too sleepy to spin up a fresh trpc server to provide it The one I mentioned before: updating your ctx||input variable to include the incoming files from the request into the context, which can then be accessed like ctx.prisma or whatever though in the case of the ctx method, you'll have to do zod validation yourself 🤔 which I like 3-5 more LOC

Gaming

Programming

Using JSZip with Zod and tRPC