Crawlee & Apify•3y ago

How can I add dynamically JS string function into `preNavigationHooks`?

I would like to dynamically add a string (which describes a JS function) to preNavigationHooks array in CheerioCrawlerOptions [1]

const crawlerOptions = {
...  preNavigationHooks: [],
});
const jsFunction = "async ({ page, request }) => { log.info(`preNavigationHook ${request.url}`); }";
crawlerOptions.preNavigationHooks.push( ??? WHAT ???) 
const myCrawler = new CheerioCrawler(crawlerOptions);

const crawlerOptions = {
...  preNavigationHooks: [],
});
const jsFunction = "async ({ page, request }) => { log.info(`preNavigationHook ${request.url}`); }";
crawlerOptions.preNavigationHooks.push( ??? WHAT ???) 
const myCrawler = new CheerioCrawler(crawlerOptions);

If I do crawlerOptions.preNavigationHooks.push(jsFunction);, when I run crawler, I got error:

WARN CheerioCrawler: Reclaiming failed request back to the list or queue. TypeError: hook is not a function at CheerioCrawler._executeHooks (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\basic\internals\basic-crawler.js:834:23) at CheerioCrawler._handleNavigation (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\http\internals\http-crawler.js:326:20) at CheerioCrawler._runRequestHandler (D:\Developpement\NodeJS\Nowis_Scraper\node_modules@crawlee\http\internals\http-crawler.js:286:24)

[1] https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#preNavigationHooks

CheerioCrawlerOptions | API | Crawlee

3 Replies

fair-rose•3y ago

you can try push(eval(jsFunction))

broad-brownOP•3y ago

Thanks for your suggestion. But eval can be harmful. I was inspired by _runHookWithEnhancedContext [1] to make my own following JS function.

  _runHookWithEnhancedContext(hooks) {
    return hooks.map((hook) => function enhancedContext(ctx) {
      const { customData } = this.input;
      hook({ ...ctx, customData });
    }.bind(this));
 }

  _runHookWithEnhancedContext(hooks) {
    return hooks.map((hook) => function enhancedContext(ctx) {
      const { customData } = this.input;
      hook({ ...ctx, customData });
    }.bind(this));
 }

And do this:

const preNavigationHooks = "[ async (context) => { context.log.info(context.customData);\n context.log.info(`preNavigationHook ${context.request.url}`); }, ]"
// Transforms a pre/post navigation hooks string into array of Functions.
evaledPreNavigationHooks = evalArrayJSFunction(preNavigationHooks, 'preNavigationHooks');
preNavigationHooks.push(..._runHookWithEnhancedContext(evaledPreNavigationHooks));

const preNavigationHooks = "[ async (context) => { context.log.info(context.customData);\n context.log.info(`preNavigationHook ${context.request.url}`); }, ]"
// Transforms a pre/post navigation hooks string into array of Functions.
evaledPreNavigationHooks = evalArrayJSFunction(preNavigationHooks, 'preNavigationHooks');
preNavigationHooks.push(..._runHookWithEnhancedContext(evaledPreNavigationHooks));

[1] https://github.com/apify/apify-sdk-js/blob/master/packages/actor-scraper/cheerio-scraper/src/internals/crawler_setup.ts#L236

GitHub

apify-sdk-js/crawler_setup.ts at master · apify/apify-sdk-js

Apify SDK monorepo. Contribute to apify/apify-sdk-js development by creating an account on GitHub.

absent-sapphire•3y ago

I guess in this case it would be better to simply extend the Crawler class

Gaming

Programming

How can I add dynamically JS string function into `preNavigationHooks`?

Did you find this page helpful?