APIFY PROXY ECONNREFUSED 127.0.0.1:80
Hello, I'm in trouble with Apify Proxy (on a free plan) I have a connexion refused. With and witout cloudscraper.
Any ideas please?
let proxyConfiguration;
if (useApifyProxy) {
proxyConfiguration = await Actor.createProxyConfiguration({ useApifyProxy: true });
} else if (proxyUrls.length > 0) {
proxyConfiguration = await Actor.createProxyConfiguration({ proxyUrls });
}
console.log(proxyConfiguration, proxyConfiguration.newUrl());
const crawler = new PlaywrightCrawler({
requestQueue,
launchContext: {
launchOptions: {
proxy: proxyConfiguration ? proxyConfiguration.newUrl() : undefined,
},
},
requestHandler: async ({ page, request }) => {
try {
// Use cloudscraper to fetch the page content
const html = await cloudscraper.get({
uri: request.url,
proxy: proxyConfiguration ? proxyConfiguration.newUrl() : undefined,
// Add other cloudscraper options as needed
});
// Process the HTML content as needed
const title = await page.title();
console.log("Page Title: " + title);
(...)
7 Replies
optimistic-goldOP•12mo ago
Failed to fetch data: Error: tunneling socket could not be established, cause=connect ECONNREFUSED 127.0.0.1:80
Hello! Is the error coming from Apify or cloudscraper?
optimistic-goldOP•12mo ago
Hi! Directly from the crawler apparently : Failed to fetch data: Error: tunneling socket could not be established, cause=connect ECONNREFUSED 127.0.0.1:80
2024-06-14T07:30:41.048Z WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Error: tunneling socket could not be established, cause=connect ECONNREFUSED 127.0.0.1:80
2024-06-14T07:30:41.050Z
groups: [],
2024-06-14T07:30:36.531Z countryCode: undefined,
2024-06-14T07:30:36.532Z password: '*',
2024-06-14T07:30:36.534Z hostname: '10.0.44.225',
2024-06-14T07:30:36.535Z port: 8011,
2024-06-14T07:30:36.537Z usesApifyProxy: true
The error is caused by the fact that
proxyConfiguration.newUrl() returns a promise, so you must await it, like this:
Moreover, when you configure the PlaywrightCrawler, there is no need to put your proxy into launchContext.launchOptions: you can just pass the proxyConfiguration object, like this:
Hope this helps! 🙂optimistic-goldOP•12mo ago
Thank you Marco, it works but I get a 403 from Cloudflare (Attention Required! | Cloudflare) with : const crawler = new PlaywrightCrawler({
requestQueue,
proxyConfiguration,
// launchContext: {
// launchOptions: {
// proxy: proxyConfiguration ? await proxyConfiguration.newUrl() : undefined,
// },
// },
requestHandler: async ({ page, request }) => {
if (dev) await page.setViewportSize({ width: 1200, height: 800 });
console.log(Processing ${request.url}...);
try {
// Use cloudscraper to fetch the page content
const html = await cloudscraper.get({
uri: request.url,
proxy: proxyConfiguration ? await proxyConfiguration.newUrl() : undefined,
// Add other cloudscraper options as needed
});
@John just advanced to level 1! Thanks for your contributions! 🎉
That depends from the anti-scraping policies the website you're interested in may have put in place, and it's a complicated matter. I suggest you to take a look at this: https://docs.apify.com/academy/anti-scraping
Anti-scraping protections | Academy | Apify Documentation
Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues.