CA
Crawlee & Apify•2y ago
crude-lavender

Proxy authentication bug?

Was trying to connect residential proxies to my crawler after using some other proxies and after changing ProxyConfiguration URLs I got error
net::ERR_TUNNEL_CONNECTION_FAILED
After checking credentials, and all possible whitelists and blacklists I realized that problem is not on my side. So I tried to implement proxy authentication in old way and it worked. This code doesn't work: 
import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:[email protected]:7000'],
  }),
});

await crawler.run(startUrls);
 
import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';

import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls:   ['http://MyUserName:[email protected]:7000'],
  }),
});

await crawler.run(startUrls);
This code works: 
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);
import { PuppeteerCrawler } from 'crawlee';
import { router } from './routes.js';

const startUrls = ['https://crawlee.dev'];

const crawler = new PuppeteerCrawler({
  requestHandler: router,
  launchContext: {
    launchOptions: {
      headless: false,
      args: ['--proxy-server=http://gate.smartproxy.com:7000'],
    },
  },
  preNavigationHooks: [
    async ({ request, page }, gotoOptions) => {
      await page.authenticate({ username: 'MyUserName', password: 'MyPassword' })
    },
  ],
});

await crawler.run(startUrls);
Should I create a bug report?
4 Replies
NeoNomade
NeoNomade•2y ago
nope it's not a bug. there are proxy providers that require a different approach sometimes for example a session id. I'm using a proxy provider that has the same behaviour, check their documentation for session id or for other ways to compose the url.
crude-lavender
crude-lavenderOP•2y ago
@NeoNomade I used bright data and they also require basic authentication and It wasn't required to authenticate manually with .authenticate I thought that crawlee handles it automatically with launch context depending on proxyUrl content. But I didn't find where exactly crawlee uses URLS to authenticate only validation and initialization.
MEE6
MEE6•2y ago
@5y just advanced to level 1! Thanks for your contributions! 🎉
NeoNomade
NeoNomade•2y ago
You don’t have to Authenticate manually. You have to generate URLs with random session ids based on their logic. I have a function for that I’ll share in a few minutes, let me reach my computer

Did you find this page helpful?