CA
Crawlee & Apify•3y ago
stormy-gold

Bug when using Apify.utils.requestAsBrowser: process seems to crash on timeout from the apify proxy

My client recently contacted me that their old apify actor does not work anymore. Upon changing the code to use the new node docker image and updating apify version a little bit everything seemed to resolve for me locally. But when on apify platform we started testing the actor with apify proxy turned on, from time to time we got the error attached in the screenshot. As I deduced, the issue seems to be coming from Apify.utils.requestAsBrowser -> http2-wrapper (source: https://github.com/szmarczak/http2-wrapper/issues/66). But I have got no idea on how to circumvent the error as we can not (?) simply use the inner libraries 😉 Relevant code snippet: 
response = await Apify.utils.requestAsBrowser({
                url: url,
                headers: {
                    'user-agent': session.userAgent,
                    'accept': 'application/json',
                    ...session.getHeaders()
                },
                proxyUrl: session.getProxyUrl()
            });
response = await Apify.utils.requestAsBrowser({
                url: url,
                headers: {
                    'user-agent': session.userAgent,
                    'accept': 'application/json',
                    ...session.getHeaders()
                },
                proxyUrl: session.getProxyUrl()
            });
GitHub
Issues · szmarczak/http2-wrapper
Use HTTP/2 the same way like HTTP/1. Contribute to szmarczak/http2-wrapper development by creating an account on GitHub.
No description
4 Replies
MEE6
MEE6•3y ago
@Zyberg just advanced to level 3! Thanks for your contributions! 🎉
Alexey Udovydchenko
Alexey Udovydchenko•3y ago
http request might be blocked when you doing it from cloud or its just random timeout because of network. add try-catch along with few retries
stormy-gold
stormy-goldOP•3y ago
Well, we have a try catch around the http request. However, the actor still restarts. So, as per ideas from the linked github issue it seems that the whole process breaks.
Alexey Udovydchenko
Alexey Udovydchenko•3y ago
Then you need to consider upgrade to recent SDK, otherwise fixing issue might require more efforts than SDK update.

Did you find this page helpful?