CA
Crawlee & Apify•2y ago
exotic-emerald

What does produce this error?

page.goto: net::ERR_TIMED_OUT does it mean the pge is blocking me? If I open in browser it works fine
5 Replies
extended-salmon
extended-salmon•2y ago
Generally speaking it does not mean that the page is blocking you. It could be just some network issues. But on the other hand - it also could be that this particular IP is blocked on this website.
exotic-emerald
exotic-emeraldOP•2y ago
I have figured out that I been blocked, but it's not the IP its some fingerprint when I'm using crawlee if I use playwrright stealth without crawlee it works fine, any idea what could be wrong here is crawlee + playwright-extra and puppeteer stealth compatible?
extended-salmon
extended-salmon•2y ago
Fingerprints are a set of headers and you could always switch them off. And although it should find suiteable fingerprint for the OS/browser - there still might be some mismatches/edge cases. To make sure it's fingerprint related - you could try adding browserPoolOptions: { useFingerprints: false }, to the crwaler options. Otherwise - crawlee works perfectly well with extra/stealth plugins: https://crawlee.dev/docs/examples/crawler-plugins
Using puppeteer-extra and playwright-extra | Crawlee
puppeteer-extra and playwright-extra are community-built
exotic-emerald
exotic-emeraldOP•2y ago
This is how my options looks right now 
{
const crawler = new PlaywrightCrawler({
  browseroolOptions: { useFingerprints: false },
  launchContext: {
    launcher: chromium.use(stealth()),
    launchOptions: {
      headless: false,
    },
  },
  requestHandler....
});
{
const crawler = new PlaywrightCrawler({
  browseroolOptions: { useFingerprints: false },
  launchContext: {
    launcher: chromium.use(stealth()),
    launchOptions: {
      headless: false,
    },
  },
  requestHandler....
});
While this one loads correctly, both files are on same project so deps are the same version `` import { chromium } from "playwright-extra"; import stealth from "puppeteer-extra-plugin-stealth"; 
chromium.use(stealth());

// That's it, the rest is playwright usage as normal 😊
chromium.launch({ headless: false }).then(async (browser) => {
  const page = await browser.newPage();
  let url = JSON.parse((await fs.readFile("./links2.json")).toString())[0];
  console.log("Testing the stealth plugin..");
  console.log(url);
  await page.goto(url, { waitUntil: "networkidle" });
});
chromium.use(stealth());

// That's it, the rest is playwright usage as normal 😊
chromium.launch({ headless: false }).then(async (browser) => {
  const page = await browser.newPage();
  let url = JSON.parse((await fs.readFile("./links2.json")).toString())[0];
  console.log("Testing the stealth plugin..");
  console.log(url);
  await page.goto(url, { waitUntil: "networkidle" });
});
foreign-sapphire
foreign-sapphire•2y ago
My proxy provider doesn't use user/pass, but IP authentication. Is it possible to get a list of the Apify outbound IPs for my proxy folks?

Did you find this page helpful?