Crawlee & Apify•3y ago

Configure Apify Proxy urls in a Crawlee Playwright crawler

I've been trying to use Apify proxies in my Crawlee crawler, but have had no luck with it, always getting an net::ERR_TUNNEL_CONNECTION_FAILED error. Evidently I'm doing something wrong, but the documentation has been extremely unhelpful, and it's been very hard to find tutorials on the matter. The relevant sections of code are as follows:

import {
  PlaywrightCrawler,
  ProxyConfiguration,
} from 'crawlee';

const proxyUrls = [
  `http://groups-BUYPROXIES94952:${this.apifyToken}@proxy.apify.com:8000`,
  `http://groups-RESIDENTIAL:${this.apifyToken}@proxy.apify.com:8000`,
];

const crawler = new PlaywrightCrawler({
  requestHandler: router,
  maxConcurrency: 10,
  navigationTimeoutSecs: 180,
  requestHandlerTimeoutSecs: 180,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls,
  }),
});

import {
  PlaywrightCrawler,
  ProxyConfiguration,
} from 'crawlee';

const proxyUrls = [
  `http://groups-BUYPROXIES94952:${this.apifyToken}@proxy.apify.com:8000`,
  `http://groups-RESIDENTIAL:${this.apifyToken}@proxy.apify.com:8000`,
];

const crawler = new PlaywrightCrawler({
  requestHandler: router,
  maxConcurrency: 10,
  navigationTimeoutSecs: 180,
  requestHandlerTimeoutSecs: 180,
  proxyConfiguration: new ProxyConfiguration({
    proxyUrls,
  }),
});

I really need some help with this. Thanks in advance.

6 Replies

vicious-gold•3y ago

what is apify token? you need to use apify proxy token

harsh-harlequin•3y ago

Maybe try to change proxyUrls to proxyUrls: proxyUrls

flat-fuchsiaOP•3y ago

Yes, that's what I'm using. The apifyToken variable contains the Apify proxy token taken from the .env. I've also already made sure that the Apify proxy token is being corretly retrieved from the .env.

vicious-gold•3y ago

then it looks good to me, do you have paid subscription? on free account external use of proxy is not allowed

flat-fuchsiaOP•3y ago

Well, then that's where the issue lies, haha. I'll change my plan and try again, thanks.

harsh-harlequin•3y ago

1) yes - external usage is available only for paid accounts. 2) apifyToken is wrong there - the should be apify proxy password - see here: https://docs.apify.com/platform/proxy/connection-settings 3) RESIDENTIAL proxy cannot be used together with datacenter proxies. Although by using proxyConfiguration from proxy and using direct proxys URLs - you skip validation completely - so it should work, but be aware, that you will randomly have either residential or datacenter proxy. 4) Finally - the best way to use Apify proxy is via Apify SDK and logging into apify in cli - that way you could do Actor.createProxyConfiguration() and just specify proxy groups there: https://docs.apify.com/sdk/js/docs/guides/apify-platform#log-in-with-cli + https://docs.apify.com/sdk/js/reference/class/Actor#createProxyConfiguration

Gaming

Programming

Configure Apify Proxy urls in a Crawlee Playwright crawler

Did you find this page helpful?