Unable to crawl
Hi, I'm trying to crawl a few pages, with this script:
Now, I don't know if I'm doing something wrong, but I'm getting timeouts and not crawling at all sometimes. The script will run, but not log any pdf, or link, it just says it crawled the two pages, and that's it. Someone, please help.
2 Replies
statutory-emerald•2y ago
Hi. You can start by running the crawler with
headless: false to see what's happening on the page. Also I see you are navigating to the request url in the request handler function, you can remove that as it is done automatically by the crawlernational-goldOP•2y ago
Hey man, thanks. I found out that the sites I want to scrape are totally fine with CheerioCrawler, it was easier that way. I'm still finding my way with the amount of options (I don't know what most them do tbh)