System architecture for processing data from multiple 3rd party APIs
Hello, I need some advice on the best practices and general direction I should consider for handling data for my application.
The current data flow is outlined in the attached image.
Issues I have with it:
1. NextJS/Vercel Serverless functions can only run for so long. Thus, I return
2. NextJS Api routes should generally only be used by the frontend, not for querying from outside of the system. Separations of concerns suggests to me that I should have another service for handling my data.
3. No queue, so if two http requests towards the same endpoint happens at the same time I'm not even sure what happens, or how I could monitor that my data, and especially my articles, are being inserted correctly.
Possible solutions:
- Building a service to handle the data using:
- AWS SQS (1 million requests for free)
- Google Cloud Computing
- Using Quirrel
- Queue system
- Cron jobs
- Self-hosted (Not sure about cost)
Concerns:
- I would like to share types between all systems. Especially those that are generated by Prisma, my zod schemas and possibly my tRPC client. Is this feasible with separated services?
- Am I thinking about this the wrong way? Am I missing something?
The current data flow is outlined in the attached image.
Issues I have with it:
1. NextJS/Vercel Serverless functions can only run for so long. Thus, I return
res.status(200).json(message) before the processes I want to run are actually finished.2. NextJS Api routes should generally only be used by the frontend, not for querying from outside of the system. Separations of concerns suggests to me that I should have another service for handling my data.
3. No queue, so if two http requests towards the same endpoint happens at the same time I'm not even sure what happens, or how I could monitor that my data, and especially my articles, are being inserted correctly.
Possible solutions:
- Building a service to handle the data using:
- AWS SQS (1 million requests for free)
- Google Cloud Computing
- Using Quirrel
- Queue system
- Cron jobs
- Self-hosted (Not sure about cost)
Concerns:
- I would like to share types between all systems. Especially those that are generated by Prisma, my zod schemas and possibly my tRPC client. Is this feasible with separated services?
- Am I thinking about this the wrong way? Am I missing something?
