CA
Crawlee & Apify•3y ago
yelping-magenta

puppeteer/replay

Hi, all, nice to see you. I have a question about how I would use puppeteer/replay in PuppeteerCrawler for executing scenario generated on Chrome Recorder. What I want to do is that, before executing crawling, function is called on each scenario step. I tried it, not using PuppeteerCrawler. But I'm not sure how it is integrated with PuppeteerCrawler. Below is a current simple example. Is there anyone who has any ideas? Thank you. 
import { BrowserPool, PuppeteerPlugin } from "@crawlee/browser-pool";
import ScenarioReplay from "./ScenarioReplay";
import fs from "fs";

// Read chrome recording json.
const recordingText = fs.readFileSync("path/to/scenario/json", "utf8");
const recording = parse(JSON.parse(recordingText));

const browserPool = new BrowserPool({
  prePageCloseHooks: [
    async (page, browserController) => {
      // do scraping and operate something on browser
    },
  ],
});

const page = await browserPool.newPage();

// create runner
const runner = await createRunner(
  recording,
  new ScenarioReplay(browserPool, page)
);

await runner.run();
await page.close();
await browser.newPage();
await browserPool.destroy();
import { BrowserPool, PuppeteerPlugin } from "@crawlee/browser-pool";
import ScenarioReplay from "./ScenarioReplay";
import fs from "fs";

// Read chrome recording json.
const recordingText = fs.readFileSync("path/to/scenario/json", "utf8");
const recording = parse(JSON.parse(recordingText));

const browserPool = new BrowserPool({
  prePageCloseHooks: [
    async (page, browserController) => {
      // do scraping and operate something on browser
    },
  ],
});

const page = await browserPool.newPage();

// create runner
const runner = await createRunner(
  recording,
  new ScenarioReplay(browserPool, page)
);

await runner.run();
await page.close();
await browser.newPage();
await browserPool.destroy();
import { PuppeteerRunnerExtension, Step, UserFlow } from "@puppeteer/replay";

class ScenarioReplay extends PuppeteerRunnerExtension {
  constructor(browser: Browser, page: Page) {
    super(browser, page);
  }

  override async beforeAllSteps(flow: UserFlow | undefined) {
    // doing something beforeAllSteps
  }

  override async beforeEachStep(step: Step, flow: UserFlow | undefined) {
    // doing something beforeEachStep
  }

  override async afterEachStep(step: Step, flow: UserFlow | undefined) {
    // doing something afterEachStep
  }

  override async afterAllSteps(flow: UserFlow | undefined) {
    // doing something afterAllSteps
  }
}

export default ScenarioReplay;
import { PuppeteerRunnerExtension, Step, UserFlow } from "@puppeteer/replay";

class ScenarioReplay extends PuppeteerRunnerExtension {
  constructor(browser: Browser, page: Page) {
    super(browser, page);
  }

  override async beforeAllSteps(flow: UserFlow | undefined) {
    // doing something beforeAllSteps
  }

  override async beforeEachStep(step: Step, flow: UserFlow | undefined) {
    // doing something beforeEachStep
  }

  override async afterEachStep(step: Step, flow: UserFlow | undefined) {
    // doing something afterEachStep
  }

  override async afterAllSteps(flow: UserFlow | undefined) {
    // doing something afterAllSteps
  }
}

export default ScenarioReplay;
5 Replies
continuing-cyan
continuing-cyan•3y ago
What is LoginScenarioReplay(...)? You export recorder script as JSON File?
yelping-magenta
yelping-magentaOP•3y ago
Sorry, I forgot to rename it😢 I have changed.
continuing-cyan
continuing-cyan•3y ago
What do you mean by "before executing the crawling, the function is called at each step of the scenario". Which function should be called?
yelping-magenta
yelping-magentaOP•3y ago
I mean “right before crawling each path, the scenario should be executed” like “beforeEachStep” of puppeteer/replay. I couldn’t find a way to do so with PuppeteerCrawler. That’s the best to execute authenticating and crawling in each browser that has Basic Auth.
automatic-azure
automatic-azure•3y ago

Did you find this page helpful?