CA
Crawlee & Apify•2y ago
harsh-harlequin

Added "playwright-extra" with "stealthPlugin" and got error "Cannot read properties of undefined"

I have some code using PlaywrightCrawler. I added "playwright-extra" with "stealthPlugin" to this code. Exactly as in documentation [1] I added to my code only this: 
import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());
import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());
The rest of program remains the same as before. And I have useFingerprints: true and launcher: firefox in code. Well, the code works. Bot detection sites report that my crawler has 3 plugins and supports 4 mime types, so something changed. But! I got this is the stdout: 
INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
INFO  PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:161:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
How bad is this? [1] https://crawlee.dev/docs/examples/crawler-plugins
Using puppeteer-extra and playwright-extra | Crawlee
puppeteer-extra and playwright-extra are community-built
8 Replies
harsh-harlequin
harsh-harlequinOP•17mo ago
Is it something related to versions of libraries? here the output of npm ls: 
npm ls
...
├── @apify/[email protected]
npm ls
...
├── @apify/[email protected]
pls help, this exception still there...
adverse-sapphire
adverse-sapphire•17mo ago
You are mixing Playwright and Puppeteer, use just one
vicious-gold
vicious-gold•16mo ago
Getting the same error using playwright-extra but doesnt really effect anything, everything continues to work as normal
other-emerald
other-emerald•16mo ago
I think puppeteer-extra plugin simply does not work with firefox. As puppeteer is chrome only, this works with playwright - chrome setting. this works without errors 
import { PlaywrightCrawler } from 'crawlee';
import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';
import { router } from './routes.js';

await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({groups:['RESIDENTIAL'], countryCode:'US'});
const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    headless:false,
    requestHandler: router,
    launchContext: {
        launcher: chromium.use(stealth())}
});
.....
import { PlaywrightCrawler } from 'crawlee';
import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';
import { router } from './routes.js';

await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({groups:['RESIDENTIAL'], countryCode:'US'});
const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    headless:false,
    requestHandler: router,
    launchContext: {
        launcher: chromium.use(stealth())}
});
.....
But sadly still blocked on sites like g2.com or bigspy.com but could help on some other sites.
vicious-gold
vicious-gold•16mo ago
Yup its only effecting firefox https://github.com/berstend/puppeteer-extra/issues/576
GitHub
[Bug] stealth plugin on Firefox throw `TypeError: Cannot read prope...
Steps to reproduce: in a new node project, run PUPPETEER_PRODUCT=firefox npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth run the following script: import puppeteer from 'pu...
vicious-gold
vicious-gold•16mo ago
It looks like I have to use the stealth plugin in order to bypass cloudflare captchas using firefox. I thought crawlee already implemented all of the features from the stealth plugin but it wont bypass cloudflare cpatchas on firefox without it. What does the stealth plugin have that crawlee doesnt?
exotic-emerald
exotic-emerald•16mo ago
I'd also like to know this! I'm fully blocked on Indeed.com regardless of implimentation.
harsh-harlequin
harsh-harlequinOP•7mo ago
Well (almost a year later, I admit it), I did some clean-up in package.json (no puppeteer any more): 
$ npm ls
...
├── @apify/[email protected]
├── @types/[email protected]
├── @types/[email protected]
$ npm ls
...
├── @apify/[email protected]
├── @types/[email protected]
├── @types/[email protected]
And I do this in code: firefox.use(stealthPlugin()); And it does not work, same error as before: 
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
    at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
Just in case all this stuff is still relevant - would be great to fix it. Would you pls look @Lukas Krivka ?

Did you find this page helpful?