Crawlee & Apify•7mo ago

Max redirects

I am getting this error message, how to best deal with it? Reclaiming failed request back to the list or queue. Redirected 10 times. Aborting. Can I increase the max number of redirects for my CheerioCrawler?

2 Replies

Hall•7mo ago

View post on community site

This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.

Apify Community

stormy-gold•6mo ago

You don’t need to do anything with it. This usually means the enqueued URL redirects somewhere else (likely a different domain), so you wouldn’t get the desired data anyway. Crawlee automatically stops processing such URLs. Also, just in case, there’s an option called followRedirect that might be helpful depending on your use case. It allows you to control whether redirects should be followed or not: https://crawlee.dev/api/next/core/interface/HttpRequest#followRedirect usage example:

const response = await gotScraping({
                url: 'https://www.example.com',
                method: 'POST',
                proxyUrl,
                body: `some_payload`,
                followRedirect: false
            }

const response = await gotScraping({
                url: 'https://www.example.com',
                method: 'POST',
                proxyUrl,
                body: `some_payload`,
                followRedirect: false
            }

Gaming

Programming

Max redirects

Did you find this page helpful?