File Uploading

Trying to implement uploading files for my app. Has anyone tried doing this with wasp? Is there a good way to do this within wasp? Any good libraries? Everything seems to go through multer, however I don’t see how we can add express middleware through wasp file or pass through headers to actions. Unless I am missing something?
17 Replies
martinsos
martinsos2y ago
Hey @entjustdoit ! So there are two ways right now to write some code on the server: 1. Via Operations (Actions / Queries) 2. Via app.server.setupFn . Approach #1 is not flexible enough for now to allow playing with headers. Approach #2 gives much more flexibility, but it doesn't currently give access to express app, so you can't attach routes to it from there. Which means we can't do this with Wasp at the moment, although we have plans to add this, and will probably do it very soon, in the following month(s). Here is an issue for it: https://github.com/wasp-lang/wasp/issues/268 . That said, I had implemented file uploading in the past, in JS / Express / React apps, and you don't really want it to go through your server -> that is a lot of traffic that you rather want to avoid, as it suffocates your server and reduces its capability to do other, more important stuff, like answer user HTTP requests. Instead, what you ideally want is to upload the file directly from client! So instead of sending it from client to server and then from server to file storage, you want to send it directly to file storage -> skip the server part. Server still plays its role, but that role becomes doing authentication / giving permissions to client to upload to file storage, instead of doing the upload itself. I guess the first question is, where do you want to store your files? Ideally you would use some file storage, like S3, or a similar offering from other hosting provider. Then, you ideally want to find a nice React library that does most of this work for you, and also explains all the details around it. Such library will probably also advise you on how to set up your S3, and what you need to do on the server. And on the server, you will most likely need to implement a route or two that do some authentication with S3 and then provide those tokens to the client (React).
GitHub
Allow defining custom API routes (http) · Issue #268 · wasp-lang/wa...
Right now dev can define operations (actions, queries) which are then consumed from client via RPC that works via http. However, they can't define custom http API roues at the moment! They ...
martinsos
martinsos2y ago
As for file uploading libraries: 1. I found this one by doing quick search, looks pretty good at first, but then it doesn't have many stars and hasn't had much work on it recently: https://github.com/apideck-samples/file-picker 2. One I used some time ago and that worked well was Fine Uploader: https://github.com/FineUploader/react-fine-uploader -> however I see now it is archived! That sucks. 3. There is this one: https://github.com/odysseyscience/react-s3-uploader -> has ok number of stars, but no work done in the last 2 years. But maybe it is stable? 4. There is this link then, pretty fresh (2022) that covers the process in details, so this might be quite interesting: https://blog.devgenius.io/upload-files-to-amazon-s3-from-a-react-frontend-fbd8f0b26f5 . It might be enough to just follow this.
martinsos
martinsos2y ago
This would be a cool feature for Wasp -> uploading of files -> and we actually have an issue for it here: https://github.com/wasp-lang/wasp/issues/494 .
GitHub
Ability to serve and upload dynamic binary assets (images, pdfs) · ...
Wasp currently has no way to let users download dynamic binary assets nor upload them. This is a fairly common use case, so should be supported. Where these assets are stored is a related topic (da...
entjustdoit
entjustdoit2y ago
❤️❤️ thank you for the thorough response! Yeah uploading to S3. I initially looked into the option of using signed uploads and skipping my server altogether as mentioned, but all the tutorials I saw kept having security disclaimers which made me nervous. I primarily wanted to make sure I wasn’t missing anything, and thought since it wasn’t asked before on here would be a good thing to clarify. I’ll dig into those links in the morning, thanks again!
martinsos
martinsos2y ago
Sure, feel free to ask! This is useful for us also, this helps us refine this topic for the future, so when we implement it we will have a lot of the stuff ready already! Ah yes, signed uploads, that is the name! Security disclaimers -> I wasn't aware of those, couple of years ago when I was doing it, most of the content online was advising to do signed uploads instead of going through our server. What were some security disclaimers they mentioned? Btw if you get stuck with anything, feel free to ask for help here, even though it is not Wasp -> as I said, it is good for us to learn about this also.
shayne
shayne2y ago
Didn't go in depth above, but signed uploads caught my eye. I, too, read they were the recommended approach, so curious what security concerns there are. If I remember correct (and it's the same thing, could be diff but I think same), your client asks your server for effectively a time-limited, one-time upload token, your server does direct AWS communication to get it (where it specifies S3 bucket too), gives the client the token, and the client can use to PUT directly to some AWS URL. I felt it was pretty safe that way? 🤷‍♂️
Vinny (@Wasp)
Vinny (@Wasp)2y ago
Yep. Here’s a great little tutorial that shows you how to do it @entjustdoit https://youtu.be/wbNyipJw9rI
TomDoesTech
YouTube
Use Presigned PUT URLs to Easily Upload Files to AWS S3
Theo's Tweet: https://twitter.com/t3dotgg/status/1564019039877271552 Repository: https://github.com/TomDoesTech/pre-signed-put-url 0:00 Intro 0:43 Data flow 1:45 Bootstrap application 2:33 S3 bucket setup 5:07 Get PUT URL handler 10:36 Form 15:24 Does it work? 🌎 Follow me here: Discord: https://discord.gg/4ae2Esm6P7 Twitter: https://twitter.c...
entjustdoit
entjustdoit2y ago
Implemented and deployed 🚀 thanks for the help! The security thing was in reference to all the tutorials I was finding and less a comment on the method itself. They kept using fully public buckets for simplifying the tutorials which is what initially turned me off the approach.
Vinny (@Wasp)
Vinny (@Wasp)2y ago
awesome! What solution did you end up using?
martinsos
martinsos2y ago
Ok yeah, that is common pitfall -> buckets need to have proper settings on them. If I remember, you can even put some CORS stuff on them? Not sure though hm. Yeah, what did you end up with Elis?
entjustdoit
entjustdoit2y ago
@Vinny (@Wasp) @martinsos Pre signed URLs for S3, just made everything more restricted in the permissions panel (eg source on CORS)
matijash
matijash2y ago
wow that was fast! 😄 any way we could make it even easier through Wasp, any ideas?
maksym36ua
maksym36ua2y ago
Hey @entjustdoit any tips or code snippets on how it can be implemented? Will appreciate 🙂
entjustdoit
entjustdoit2y ago
Sure! First, go to S3 and setup a bucket and IAM user, then add these entries to your .env.server file.
AWS_S3_IAM_ACCESS_KEY=
AWS_S3_IAM_SECRET_KEY=
AWS_S3_FILES_BUCKET=
AWS_S3_REGION=
AWS_S3_IAM_ACCESS_KEY=
AWS_S3_IAM_SECRET_KEY=
AWS_S3_FILES_BUCKET=
AWS_S3_REGION=
then add the aws sdk to your list of dependencies
dependencies: [
...
("aws-sdk", "^2.1294.0"),
...
]
dependencies: [
...
("aws-sdk", "^2.1294.0"),
...
]
In your front end, setup the functions for downloading and uploading files.
const handleUploadFile = async () => {
...
let data = await getUploadFileSignedURL(...);
// key is the identifier of the file in S3
const { uploadUrl, key } = data;
// upload the actual file using the signed URL, newFile here is the file selected in the form
await axios.put(uploadUrl, newFile);

// store the key as a field of a file entity for later retrieval
...
}
const handleUploadFile = async () => {
...
let data = await getUploadFileSignedURL(...);
// key is the identifier of the file in S3
const { uploadUrl, key } = data;
// upload the actual file using the signed URL, newFile here is the file selected in the form
await axios.put(uploadUrl, newFile);

// store the key as a field of a file entity for later retrieval
...
}
const handleDownloadFile = async (file) => {
...
let downloadUrl = await getDownloadFileSignedURL(...);
// ignore my ugly code below, its a workaround I had to do due to how I had setup my UI
var link = document.createElement("a");
link.download = file.filename;
link.href = downloadUrl;
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
...
}
const handleDownloadFile = async (file) => {
...
let downloadUrl = await getDownloadFileSignedURL(...);
// ignore my ugly code below, its a workaround I had to do due to how I had setup my UI
var link = document.createElement("a");
link.download = file.filename;
link.href = downloadUrl;
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
...
}
We now need to create two queries for getting the upload and download signed URLs.
...
import S3 from 'aws-sdk/clients/s3.js';

const s3Client = new S3({
apiVersion: '2006-03-01',
accessKeyId: process.env.AWS_S3_IAM_ACCESS_KEY,
secretAccessKey: process.env.AWS_S3_IAM_SECRET_KEY,
region: process.env.AWS_S3_REGION,
signatureVersion: 'v4'
})
...
export const getUploadFileSignedURL = async (args, context) => {
... // business logic for who can upload
const ex = args.fileType;
const Key = `${uuid()}.${ex}`;
const s3Params = {
Bucket: process.env.AWS_S3_FILES_BUCKET,
Key,
Expires: 30,
ContentType: `${ex}`,
};
const uploadUrl = await s3Client.getSignedUrl("putObject", s3Params);
return { uploadUrl, key: Key };
}

export const getDownloadFileSignedURL = async (args, context) => {
... // business logic for who can download
const s3Params = {
Bucket: process.env.AWS_S3_FILES_BUCKET,
Key: args.Key,
Expires: 30,
};
const downloadUrl = await s3Client.getSignedUrl("getObject", s3Params);
return downloadUrl;
}
...
import S3 from 'aws-sdk/clients/s3.js';

const s3Client = new S3({
apiVersion: '2006-03-01',
accessKeyId: process.env.AWS_S3_IAM_ACCESS_KEY,
secretAccessKey: process.env.AWS_S3_IAM_SECRET_KEY,
region: process.env.AWS_S3_REGION,
signatureVersion: 'v4'
})
...
export const getUploadFileSignedURL = async (args, context) => {
... // business logic for who can upload
const ex = args.fileType;
const Key = `${uuid()}.${ex}`;
const s3Params = {
Bucket: process.env.AWS_S3_FILES_BUCKET,
Key,
Expires: 30,
ContentType: `${ex}`,
};
const uploadUrl = await s3Client.getSignedUrl("putObject", s3Params);
return { uploadUrl, key: Key };
}

export const getDownloadFileSignedURL = async (args, context) => {
... // business logic for who can download
const s3Params = {
Bucket: process.env.AWS_S3_FILES_BUCKET,
Key: args.Key,
Expires: 30,
};
const downloadUrl = await s3Client.getSignedUrl("getObject", s3Params);
return downloadUrl;
}
martinsos
martinsos2y ago
Awesome @entjustdoit , thanks for helping out :D! Ha we have Waspeteers helping each other this is really nice :D.
maksym36ua
maksym36ua2y ago
Simply amazing, can't thank you enough! 🔥🔥🔥
entjustdoit
entjustdoit2y ago
No worries!
Want results from more Discord servers?
Add your server