CA
Crawlee & Apify•2y ago
multiple-amethyst

User agent isn't randomly created by crawlee

I'm using the PuppeteerCrawler class and my expectation is that the user agent should be randomly generated whenever I create a new instance (or start a new run) This is not what happens, instead, the same value is consistently used across runs 
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9
How do I rotate the user agent? Sometimes, the user agent just looks like my normal laptop user agent as well. Not sure why. Note, that I am also using Pupeteer extra. For reference, this is a simplified look at what my Instantion looks like. 
new PuppeteerCrawler({
    launchContext: {
      // !!! You need to specify this option to tell Crawlee to use puppeteer-extra as the launcher !!!
      launcher: puppeteerExtra,
      launchOptions: {
        // Other puppeteer options work as usual
        headless: false,
      },
    },
  })
new PuppeteerCrawler({
    launchContext: {
      // !!! You need to specify this option to tell Crawlee to use puppeteer-extra as the launcher !!!
      launcher: puppeteerExtra,
      launchOptions: {
        // Other puppeteer options work as usual
        headless: false,
      },
    },
  })
1 Reply
national-gold
national-gold•2y ago
Hi! Regarding the user agent - by default it uses the default browser and underlying operating system. To change this, please refer here to override the default fingerprint options: https://docs.apify.com/academy/anti-scraping/mitigation/generating-fingerprints https://crawlee.dev/docs/guides/avoid-blocking#customizing-browser-fingerprints
Generating fingerprints | Apify Documentation
Learn how to use two super handy NPM libraries to easily generate fingerprints and inject them into a Playwright or Puppeteer page.
Avoid getting blocked | Crawlee
How to avoid getting blocked when scraping

Did you find this page helpful?