environmental-rose

About define route

I'm using Crawlee for Crawler platform. I have a question: I can use routing for preNavigationHooks, failedRequestHandler... as requestHandler. If I can, how to use it for preNavigationHooks. Thanks!

5 Replies

HonzaS•3y ago

you need to make the logic yourself in preNavigationHooks or failedRequestHandler

Alexey Udovydchenko•3y ago

both preNav and failedReq supposed to be for all requests per crawler, if you need logic there do it by request.userData params

environmental-roseOP•3y ago

When I init PuppeteerCrawler constructor how to I push userData to request. I want to base on data user input to process parser?

HonzaS•3y ago

not in constructor but in run method this will put https://example.com/ request with label exampleLabel to the crawler request queue

await crawler.run([{url:"https://example.com/", userData:{label:"exampleLabel"}]);

await crawler.run([{url:"https://example.com/", userData:{label:"exampleLabel"}]);

environmental-roseOP•3y ago

Thank you

Gaming

Programming

About define route

Did you find this page helpful?