Best way to handle fetching large amounts of emails
So I have a T3 app that is essentially a project manager for a construction/remodel company. Every time the company is starting a project on a client's house, they create a new "project' in our app. One of the features for the scope is to have an "email" tab where we use the gmail API to fetch every conversation an employee of the company has had with the client, and display them in the app.
The gmail API for orgs does have an endpoint to fetch all emails built in, but based on the fact that my fetch for single users emails takes like 3-5 seconds ATM, I'm guessing that doing a fetch on all this data at runtime is not going to to be viable.
What approach would you guys recommend? One approach one of my coworkers floated was to essentially have a separate server that's running 24/7, and does this fetch call once every minute. We then have a DB that essentially just has one table, customers, and each customer has an array of messageIds. All we do is add in add in the ID of any message that has happened since the last fetch to the array. Then on the main app, we fetch the list of messageIds from the DB for the customer, and use that pre-compiled list of messages to fetch each email needed.
Another similar approach would be that whenever an "admin" logs in on the app, and opens a project, we do a fetch for their conversations with the given client, then push that to the main DB we already have. Then we can fetch everyone's previous convos. Downside to this approach is that the emails would only be updated when one of the admin logs in and views the projects, so recent emails could take a while to come, users don't have a way of knowing if the list of emails their looking at is stale or not.
Curious to hear thoughts about better approaches as I'm not really a big fan of either of these!
The gmail API for orgs does have an endpoint to fetch all emails built in, but based on the fact that my fetch for single users emails takes like 3-5 seconds ATM, I'm guessing that doing a fetch on all this data at runtime is not going to to be viable.
What approach would you guys recommend? One approach one of my coworkers floated was to essentially have a separate server that's running 24/7, and does this fetch call once every minute. We then have a DB that essentially just has one table, customers, and each customer has an array of messageIds. All we do is add in add in the ID of any message that has happened since the last fetch to the array. Then on the main app, we fetch the list of messageIds from the DB for the customer, and use that pre-compiled list of messages to fetch each email needed.
Another similar approach would be that whenever an "admin" logs in on the app, and opens a project, we do a fetch for their conversations with the given client, then push that to the main DB we already have. Then we can fetch everyone's previous convos. Downside to this approach is that the emails would only be updated when one of the admin logs in and views the projects, so recent emails could take a while to come, users don't have a way of knowing if the list of emails their looking at is stale or not.
Curious to hear thoughts about better approaches as I'm not really a big fan of either of these!
Was this page helpful?
