Crawlee & Apify•5mo ago

Issue with Residential Proxies

Hi there. Whenever I try to use residential proxies ('HTTP://groups-RESIDENTIAL:/...') I run into this error: httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129) The 'auto' group seems to work fine. Can anyone tell me what I'm doing wrong here? Thanks!

6 Replies

Hall•5mo ago

Someone will reply to you shortly. In the meantime, this might help: -# This post was marked as solved by uberpea5000. View answer.

helpful-purpleOP•5mo ago

Hmm. The automated response may have directed me to an answer: I'm crawling a .gov URL.

sunny-green•5mo ago

Hey Try disabling certificate validation

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import HttpxHttpClient


async def main() -> None:
    http_client = HttpxHttpClient(verify=False)
    crawler = BeautifulSoupCrawler(http_client=http_client)

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import HttpxHttpClient


async def main() -> None:
    http_client = HttpxHttpClient(verify=False)
    crawler = BeautifulSoupCrawler(http_client=http_client)

this work from version 0.5.0

helpful-purpleOP•5mo ago

Ah ha! Yes. That seems to do it. Much appreciated. (It alsoimplies the solution to my next problem: extending the httpx client timeout period.)

sunny-green•5mo ago

Note that if httpx will be very slow (more than a minute), you may also need to increase the timeout for the handler - https://crawlee.dev/python/api/class/BasicCrawlerOptions#request_handler_timeout

helpful-purpleOP•5mo ago

Thanks. Yes. Just had to do that as well 🙂 Got another question, but I'll start a new thread for it. Thanks again.

Gaming

Programming

Issue with Residential Proxies

Did you find this page helpful?