Something is off when running from Apify
Hey,
So, I developed the NewEgg scraper actor, and I’m having some trouble, particularly because I think NewEgg upped their anti-bot protection during holiday season.
However, something weird is happening, when I run the code locally using your residential proxies, it sometimes fails, but generally it doesn’t, specially after I implemented UA Rotation.
But, from Apify (and from your maintenance runs), it always fails, this kinda indicates to me that it’s not being able to properly connect to your residential proxies via the platform.
8 Replies
Someone will reply to you shortly. In the meantime, this might help:
@Rorisjack just advanced to level 2! Thanks for your contributions! 🎉
ugly-tanOP•5mo ago
Actually just tested it, proxy usage goes up via the platform,
Nonetheless the exact same code is failing when running locally vs the platform?
xenial-black•5mo ago
are you using automated browser?
ugly-tanOP•5mo ago
Nope
I’m merely scraping using httpx and then BS4 to parse the HTML
xenial-black•5mo ago
Ok, so that is weird. Pure request should be the same.
ugly-tanOP•5mo ago
right? it’s very weird
i suppose it should be temporary, i bet the extra anti-bot / anti-scraping measures are a “before christmas” thing at newegg
but still, it’s weird
xenial-black•5mo ago
I have tried this url https://www.newegg.com/asrock-rx6600-cld-8g-amd-radeon-rx-6600-8gb-gddr6/p/N82E16814930066 and it works even with simple request from insomnia