CA
Crawlee & Apify•3y ago
harsh-harlequin

Apify-web-scraper does not scrape all enqueued urls

Hi everyone, I am modifying the apify/web-scraper actor to scrape script tags from a list of urls and then enqueue other links. However, I discover that the scraper stops before scraping all of the enqueued urls without throwing an error, so it is difficult to debug what could be going wrong. Please see the pageFunction attached. Thanks for your help.

pageFunction.txt

1 Reply
manual-pink
manual-pink•3y ago
The code is fairly complex. I would just log the URLs before 
await context.enqueueRequest({ 
            url: nextUrl
        });
await context.enqueueRequest({ 
            url: nextUrl
        });
they might be duplicate

Did you find this page helpful?