CA
optimistic-gold
Scraping Threads Timeline. But Not able to get entire data.
I am making a scrapper to scrap threads.net using playwright and python and willing to host on Apify.
But It I am not able to scrap entire timeline. It is only giving some data and when scrolling down, I am not able to scrap the new data.
Any help or suggestion is highly appreciated.
3 Replies
Try https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/paginating-through-results#auto-scrolling or find site-specific controls
variable-lime•2y ago
I recommend storing screenshots often, you will see exactly where you got stuck and why. More here https://docs.apify.com/academy/node-js/analyzing-pages-and-fixing-errors
How to analyze and fix errors when scraping a website | Academy | A...
Learn how to deal with random crashes in your web-scraping and automation jobs. Find out the essentials of debugging and fixing problems in your crawlers.
optimistic-goldOP•2y ago
Thank you guys. It helped.