Crawler only working in headed mode.
I have a Puppeteer Crawler that works almost flawless in headed mode, but if I go headless all the requests are getting 403 errors.
I was thinking that xvfb should fix this but unfortunately it doesn't. Any other ideas ?
9 Replies
deep-jade•2y ago
Please share run, I guess its site specific, they using blocking of some kind
hipercor.es/supermercado
rising-crimson•2y ago
@NeoNomade did you find any solution ?
ambitious-aqua•2y ago
@NeoNomade
Can't find your actor via "hipercor.es/supermercado"
Some suggestions:
1. try to test different proxy groups
2. try to modify (change some) headers (as in original requests in the browser)
3. Try to use Playwright + firefox. It's more stealthy (btw, why are You using pptr?)
1. fixed it, I fixed it for a short time with xvfb
2. will try
3. will also try, I'm using pptr because I find it easier to code
ambitious-aqua•2y ago
3. Why do You think so?)
I would say, that Playwright is improved pptr. And for sure it's easier to bypass anti-bot protection with Playwright.
will try with that one too
coded playwright in the past but found easier the workflow in puppeteer :))
rising-crimson•2y ago
@NeoNomade how did you fix it with XVFB ?
Didn’t fix it, it worked with playwright and firefox