Crawlee & Apify•3y ago

Using requestsFromUrl is throwing an Error

When i tried fetching the URLs from a text file, Apify throwed an error with the latest crawlee/playwrightcrawler setup. I have attached a screenshot of the same error with crawlee alone running in my local machine.

2 Replies

broad-brown•3y ago

Its out of SDK, you need:

const fromStartUrls = async function* (startUrls, name = 'STARTURLS') {
    const rl = await RequestList.open(name, startUrls);

    let rq;
    // eslint-disable-next-line no-cond-assign
    while (rq = await rl.fetchNextRequest()) {
        yield rq;
    }
};

const fromStartUrls = async function* (startUrls, name = 'STARTURLS') {
    const rl = await RequestList.open(name, startUrls);

    let rq;
    // eslint-disable-next-line no-cond-assign
    while (rq = await rl.fetchNextRequest()) {
        yield rq;
    }
};

and then:

    const requestQueue = await RequestQueue.open();
    for await (const req of fromStartUrls(startUrls)) {
        await requestQueue.addRequest(req);
    }

    const requestQueue = await RequestQueue.open();
    for await (const req of fromStartUrls(startUrls)) {
        await requestQueue.addRequest(req);
    }

Explanation: when user uploading file from UI actor getting URL with uploaded file, so actual URLs must be resolved from it

extended-salmonOP•3y ago

Thank you so much. This solution worked!

Gaming

Programming

Using requestsFromUrl is throwing an Error

Did you find this page helpful?