Web scraper with a database and WebAPI + MVC - help needed
I'm working on a project which requires me to use ASP.NET Core to make a WebAPI, and also a completely seperate MVC app which consumes the WebAPI. It's a college project and these are the requirements which must be met.
To achieve this, I want to build a web scraper which scrapes job ad info for software development related jobs, and displays the demand for various technologies to users to guide them on what they should focus on learning to suit the job market.
So as far as I understand, there are three components to my project:
- Web scraper (written in C#)
- WebAPI service
- MVC application serving the front-end which uses data from the above API
What I need help with:
I do not know how to structure my project so that all of these components work together nicely. For example, I was thinking I could make a .NET console app, add Entity Framework Core and then setup a "Job" and "Technologies" models which have a many-to-many relationship. The web scraper scrapes the data and adds it to the database via EF.
But then how do I connect the WebAPI service to that same database? And how do I setup models in my WebAPI project without duplicating the models I created for the scraper? Can these projects share the same models?
Or should my WebAPI project handle all of the models and database connection, and then the scraper just sends POST requests with the scraped data to the WebAPI?
Would really appreciate any advice on how to best go about this!
1 Reply
Simplest way would be to make an MVC app, add an API surface, and add a HostedService for the scraper.
Now you have all three components in a single executable project.
But before you go ahead and do that - make sure your scraper is legal and allowed.
A lot if not most webpages today explicitly disallow webscraping