CA
Crawlee & Apify•11h ago
TimCoulter

Errors while attempting to use Apify Proxy

I have created a very simple actor that uses the gotScraping library to retrieve the HTML from a set of YouTube video pages. The page URLs are passed as an input array. 
await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();
await Actor.init();

const input = await Actor.getInput<Inputs>();
const store = await Actor.openKeyValueStore('html-page-store');

for (const url of input!.urls) {
    log.info(`Fetching page HTML for video ID: ${url}`);

    try {
        const proxyConfiguration = await Actor.createProxyConfiguration({
            groups: ['RESIDENTIAL'],
            countryCode: 'US',
        });

        const proxyUrl = await proxyConfiguration!.newUrl();

        log.info(`Using proxy URL:`, { proxyUrl });

        const requestOptions = {
            url,
            proxyUrl,
            headers: { ...Constants.defaultRequestHeaders }
        };

        const response = await gotScraping.get(requestOptions);

        await store.setValue(url, response.body, {
            contentType: 'text/html',
        });

        log.info(`Data extracted for video ID: ${url}`);
    } catch (error) {
        log.error(`Error extracting data for video ID: ${url}`, { error });
    }
}

await Actor.exit();
When the actor runs, it correctly retrieves the first page in the sequence, but crashes when attempting to retrieve a different proxy for the second page (see attached screenshot) What puzzles me is that the error does not get caught by the catch block in my own code; it seems to crash at a lower level. I am also not clear why I am seeing the warning message WARN ProxyConfiguration: Apify Proxy access check timed out. and how to correct this issue. None of the docs seem to mention it. Lastly, can you please clarify: if I want a different proxy for every request, should my call to Actor.createProxyConfiguration be inside the loop, or prior to the loop? Hoping somebody can advise. Many thanks, Tim
No description
2 Replies
Hall
Hall•11h ago
Someone will reply to you shortly. In the meantime, this might help:
azzouzana
azzouzana•8h ago
Hey 👋 You should move the proxy declaration outside of the for loop. It's by nature a rotating proxy I believe (you can ping any IP service like https://www.ipify.org/ and see its result to confirm). Also, I'm wondering why do you want to use raw gotScraping instead of the available crawlers (which should lift most of the work?
ipify - A Simple Public IP Address API
ipify API is a simple public IP address API, easy enough to integrate into any application in seconds.

Did you find this page helpful?