CA
solid-orange
proxy issues
Hello,
I am new to crawlee and I am hoping someone could help me here. Well I have built my first crawler and it's working okay without a proxy but when I add a proxy (see code attached) - the crawler works but it does not utilize the proxy. oxylabs or scrapeops proxy endpoints are not receiving any traffic. As a newbie on crawlee is there something obvious I am missing here? Both proxy providers are working okay when tested with scrapy. Please let me know if there is something I am dong wrong here and I appreciate in advance (code is attached). Thank you!
3 Replies
solid-orangeOP•14mo ago
solid-orangeOP•14mo ago
Just to give more context; I also tried to switch proxyConfiguration (instead of having it inside the launchOptions (see below). The behavior in this case is different. The code runs but i get this SSL issue
WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.goto: net::ERR_CERT_AUTHORITY_INVALID at https://www.example.com/us/therapists/79054?page=1
const crawler = new PlaywrightCrawler({
proxyConfiguration,
launchContext: {
launchOptions: {
headless: true,
browser: chromium,
},
},
the second way is the correct one how to use custom proxies. Have you tried it with browsers other than chromium and scrape other websites? Otherwise you would probably need to disable SSL errors in the browser context and launch a new browser