CA
wise-white
Rotate proxy when request is timing out
I'm utilizing a third-party proxy provider within Playwright. Whenever the page loading initiates and starts timing out, I'm aiming to remove and cycle the proxy that the browser is using. This should apply to all browser requests currently being handled or queued for processing. Unfortunately, I'm struggling to locate suitable documentation or a clear example for this. Could someone please point me towards an example or assist me in resolving this issue?
Error message: Reclaiming failed request back to the list or queue. page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://example.com
4 Replies
You need to ask proxy provider how to create or rotate sessions based on theirs proxy, SDK can not do that for custom proxies because its unknown to SDK how it needs to be done. If your proxy accpting session as URL parameter then you can create array of proxy URLs and pass it to crawler to rotate. If its something else you need to add your own logic by code to crawler.
wise-whiteOP•2y ago
@Alexey Udovydchenko So there is an API provided by the proxy provider which I can call to delete the proxies which are timing out but I am not sure where and how should I do it with Crawlee Playwright. The issue is to kill all the current request and queued with the timed out proxy.
@Rahul just advanced to level 1! Thanks for your contributions! 🎉
@Rahul in requestHandler you need to 1session.markBad()` https://crawlee.dev/docs/next/guides/session-management - it will leads to reloading pages based on the given session. Creation of your custom session can be done by https://crawlee.dev/api/next/core/interface/SessionPoolOptions or might be easier just to add preNavigationHooks and modify crawlingContext.proxyInfo.url