CA
Crawlee & Apify•3y ago
extended-salmon

Downloading an image using puppeteer example

Hi, I found this simple example using Puppeteer, that downloads images as you visit the page and I'm wonderign how I can encorporate it into my crawlee scraper. 
    this.page.on('response', async (response) => {
      const matches = /.*\.(jpg|png|svg|gif)$/.exec(response.url());

      console.log(matches);

      if (matches && (matches.length === 2)) {
        const extension = matches[1];
        const buffer = await response.buffer();

        fs.writeFileSync(`downloads/${this.request.userData}.${extension}`, buffer, 'base64');
        counter += 1;
      }
    });
    this.page.on('response', async (response) => {
      const matches = /.*\.(jpg|png|svg|gif)$/.exec(response.url());

      console.log(matches);

      if (matches && (matches.length === 2)) {
        const extension = matches[1];
        const buffer = await response.buffer();

        fs.writeFileSync(`downloads/${this.request.userData}.${extension}`, buffer, 'base64');
        counter += 1;
      }
    });
2 Replies
extended-salmon
extended-salmonOP•3y ago
It's from here, step 3 https://www.bannerbear.com/blog/how-to-download-images-from-a-website-using-puppeteer/
How to Download Images from a Website Using Puppeteer
This article will take you through steps to download images from a website using Puppeteer.
extended-salmon
extended-salmonOP•3y ago
Okay I figured it out: 
      preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
          const { page, request } = crawlingContext;

          console.log('handlePageFunction');

          page.on('response', async (response) => {
            const matches = /.*\.(jpg|png|svg|gif)$/.exec(response.url());

            if (matches && (matches.length === 2)) {
              const extension = matches[1];
              const buffer = await response.buffer();

              fs.writeFileSync(
                `downloads/${request.userData.row.id}.${extension}`, 
                buffer,
                'base64'
              );
            }
          })
        }
      ]
      preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
          const { page, request } = crawlingContext;

          console.log('handlePageFunction');

          page.on('response', async (response) => {
            const matches = /.*\.(jpg|png|svg|gif)$/.exec(response.url());

            if (matches && (matches.length === 2)) {
              const extension = matches[1];
              const buffer = await response.buffer();

              fs.writeFileSync(
                `downloads/${request.userData.row.id}.${extension}`, 
                buffer,
                'base64'
              );
            }
          })
        }
      ]

Did you find this page helpful?