Crawlee & Apify•4w ago

Proxy Support

Is this the right away to add proxy support to an Actor? I'm guessing this is using the datacenter proxies included in my plan? Would users would be able to buy their own proxies and it would work? Or does the groups need to somehow get the proxy group name they purchased?

            proxy_configuration = await Actor.create_proxy_configuration(
                groups=['BUYPROXIES94952'],
                country_code='US'
            )
            proxy_url = await proxy_configuration.new_url()
            Actor.log.info(f'Using proxy URL: {proxy_url}')

            async with AsyncClient(timeout=30.0) as client:
                transport = client._transport  
                transport.proxies = {'http': proxy_url, 'https': proxy_url}

            proxy_configuration = await Actor.create_proxy_configuration(
                groups=['BUYPROXIES94952'],
                country_code='US'
            )
            proxy_url = await proxy_configuration.new_url()
            Actor.log.info(f'Using proxy URL: {proxy_url}')

            async with AsyncClient(timeout=30.0) as client:
                transport = client._transport  
                transport.proxies = {'http': proxy_url, 'https': proxy_url}

4 Replies

Hall•4w ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by Matous. View answer.

Louis Deconinck•4w ago

I think that will work, but if you just want to use the datacenter proxies, that is the default. So you could also write it like this.

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new CheerioCrawler({
  // ...
  proxyConfiguration,
  requestHandler({ proxyInfo }) {
     const usedProxyUrl = proxyInfo.url; // Getting the proxy URL
  }
})

const proxyConfiguration = await Actor.createProxyConfiguration();

const crawler = new CheerioCrawler({
  // ...
  proxyConfiguration,
  requestHandler({ proxyInfo }) {
     const usedProxyUrl = proxyInfo.url; // Getting the proxy URL
  }
})

Documentation: https://docs.apify.com/sdk/js/reference/class/ProxyConfiguration

ProxyConfiguration | API | SDK for JavaScript | Apify Documentation

Configures connection to a proxy server with the provided options. Proxy servers are used to prevent target websites from blocking your crawlers based on IP address rate limits or blacklists. Setting proxy configuration in your crawlers automatically configures them to use the selected proxies for all connections. You can get information about t...

absent-sapphireOP•4w ago

Thanks, how do I allow any type of proxy? Is hard coding the groups going to force a type?

Louis Deconinck•4w ago

You mean both residential and data center? That's not a common use case and there's no easy way to do that. I think you would have to get all of the proxy URLs dynamically and feed them to the proxy config. Be aware that you will pay based on the GB sent through the residential proxies, so might be expensive. If you just want to allow all datacenter proxies, there's nothing to do. By not providing any groups that will happen automatically.

Gaming

Programming

Proxy Support

Did you find this page helpful?