More meaningful error than ERR_TUNNEL_CONNECTION_FAILED
Hi There.
I am using a proxy to crawl some sites and encounter a ERR_TUNNEL_CONNECTION_FAILED error.
I am using brightData as my proxy service. If i was to curl my proxy endpoint I get a meaningful errror. For example
HTTP/1.1 502 Could not resolve host demo-site.reckoniq.coms
x-brd-err-code: target_40001
x-brd-err-msg: Could not resolve host demo-site.reckoniq.coms
X-Luminati-Error: Could not resolve host demo-site.reckoniq.coms
x-brd-error: Could not resolve host demo-site.reckoniq.coms
Proxy-Connection: close
or
HTTP/1.1 403 Forbidden
x-brd-err-code: policy_20050
x-brd-err-msg: Forbidden: target site requires special permission. You are trying to access a target site which is not permitted by our compliance policy. In order to gain access you may need to undergo a KYC process, you can do so by filling in the form: https://brightdata.com/cp/kyc If you have already completed the KYC approval, please contact your account manager for further details.
X-Luminati-Error: Forbidden: target site requires special permission. Contact BrightData for assistance
x-brd-error: Forbidden: target site requires special permission. Contact BrightData for assistance
Proxy-Connection: close
I am wondering how i can surface those errors in crawlee rather than just getting
ERROR PlaywrightCrawler: Request failed and reached maximum retries. Error: Detected a session error, rotating session...
page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://demo-site.reckoniq.coms/
There is no response in the error handler so things are not makinging it that far
Thanks for any help
Bright Data - Web Data Platform
World's largest proxy service with a residential proxy network of 72M IPs worldwide and proxy management interface for zero coding. Start a 7-day free trial »
3 Replies
Someone will reply to you shortly. In the meantime, this might help:
useful-bronze•3mo ago
Hi @shansmithnz
I don't think we can get more from this error, it is what playwright gives us when it fails to establish the connection.
deep-jade•3mo ago
You can surface detailed proxy errors in Crawlee by modifying the error handling logic to capture and log specific proxy response codes like 502 or 403. To achieve this, ensure that your crawler checks for proxy-specific errors and logs the response messages, such as "Could not resolve host" or "Forbidden: target site requires special permission". Additionally, you may want to enhance your session management to handle proxy errors gracefully and provide meaningful feedback rather than a generic session error.