Stop crawler at specific request
Is it possible to stop the crawler at a specific request and leave the window open to inspect it via devtools?
When using headless:false it seems like the window is closed after the requestQueue has been processed.
It also would be nice to have the devtools: true option in the puppeteer config...
3 Replies
deep-jade•3y ago
Just add condition with
sleep() and you need to run headfull browser. Also in most of cases you should be able to reproduce same data flow manually by running regular browser in incognito mode. What you asking for is needed in very rare cases. I suggest to develop data flow in incognito browser first then copy selectors and code into scraper.exotic-emeraldOP•3y ago
@Alexey Udovydchenko thanks 🙂 I agree that these are edge cases but chromium doesn't always match the incognito browser so debugging sometimes is necessary. I'm using sleep() and headless:false right now (awaiting a promise with setTimeout) - its ok.
@Alexey Udovydchenko just found out that it's actually possible to activate devtools via
launchOptions: {
devtools: true,
},other-emerald•3y ago
@bs Didn't know that actually! That's cool