CA
Crawlee & Apify•3y ago
metropolitan-bronze

Cannot add requests to my actor requestQueue

Hello guys, this is my first post on this discord, first of all thanks in advance for the potential help ! I'm new to apify and was trying out pupeteer crawler, to get all the products informations on a website. For that, i need to loop through all the products of a given page and get all the product links to enqueue them to my requestQueue. But when I add them it doesn't seem to go through the others requests.. I uploaded as a file what I came up with as apify.js and what requestQueue object was returned. Please help, i've tried many different syntaxes but none of them seem to work out, I feel like I struggle to make a real difference between Apify, Pupeteer, Crawlee and don't know where to search. Thank you !

message.txt

apify.js

4 Replies
absent-sapphire
absent-sapphire•3y ago
Looking at the code - you're trying to add requests to the queue in browser context, while queue exists in the node context. await page.$$eval means that this code is executed inside of the browser. What you need to do is to just return the urls from the browser, and add them to queue out of browser context, somewhat like the following (I did not check it, but you'll get the idea): 
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedData = [];

    $products.forEach(($product) => {
        scrapedData.push({
            title: $product.querySelector('.c-product-list-item--grid__title').innerText,
            url: $product.querySelector('.c-product-list-item--grid__title').href,
        });
    });

    return scrapedData;
});

await crawler.addRequests(
    // Your requests here. E.g. products.map((prod) => return { url: prod.url }));
)
const products = await page.$$eval('.c-product-list-container-products__item', ($products) => {
    const scrapedData = [];

    $products.forEach(($product) => {
        scrapedData.push({
            title: $product.querySelector('.c-product-list-item--grid__title').innerText,
            url: $product.querySelector('.c-product-list-item--grid__title').href,
        });
    });

    return scrapedData;
});

await crawler.addRequests(
    // Your requests here. E.g. products.map((prod) => return { url: prod.url }));
)
generous-apricot
generous-apricot•3y ago
For beginner I would recommend going through basic tutorial - https://developers.apify.com/academy/web-scraping-for-beginners
Apify
Web scraping for beginners · Apify Developers
Learn how to develop web scrapers with this comprehensive and practical course. Go from beginner to expert, all in one place.
metropolitan-bronze
metropolitan-bronzeOP•3y ago
@Andrey Bykov Good evening ! I didn't take the time to thank you for your response, I got it working yesterday ! I didn't get the fact that the only functions that are executed in the browser are the ones in page.$eval. Thank you ! @Lukas Krivka Good evening ! Thank you for the link but I searched through the docs already otherwise I wouldn't have asked here.
absent-sapphire
absent-sapphire•3y ago
@Guigui sure thing, happy to help!

Did you find this page helpful?