Help me!
now I am scraping a website.
This website is similar to jobsite, but there are 3066 job posts. However, since it cannot load all of them at once, it imports 20 more at a time using data-page="10" in HTML. I can move the mouse down to continue receiving data. Since I use a program, I can only read the 20 with my script.
How can I treat this problem?
print(len(searchItems)) must be 3066 but It is set to 20 because it has not been loaded.
1 Reply
fair-rose•2y ago
Hey, to scrape dynamically loaded content on a website, you'd have to simulate scrolling in the crawler (or with mouse or by pressing keys events). Alternatively, you can try to inspect the incoming request in Dev Tools and scrape their API: https://docs.apify.com/academy/api-scraping
API scraping | Academy | Apify Documentation
Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.