PuppeteerCrawler navigation timeout
I am using PuppeteerCrawler. Most of the time on the 10th page I get timeout error. I have tried:
* increasing timeout to 100s => does not help
* increasing memory/CPU => 0.25 to 1 CPU => does not help
When I click the web manually in anonymous Chrome, it works just fine.
Any idea how to debug it, please? I am blind about what is happening on the 10th page...
1 Reply
extended-salmon•3y ago
You can try to make screenshot to see what is going on with the page:
https://sdk.apify.com/docs/examples/capture-screenshot
But you should do it in
preNavigationHooks (https://sdk.apify.com/docs/2.3/typedefs/puppeteer-crawler-options#prenavigationhooks).
Also, you can try to run your code locally with headless: false option + use utils.sleep() (https://sdk.apify.com/docs/2.3/api/utils#utilssleepmillis) in some points to see how your code goes.