CA
other-emerald
Fingerpring management
Hi,
I use Crawlee for a few months and I greatly appreciate it;-)
I face bot detection problem by Imperva for a few days.
Solutions such as Zenrows or Brightdata are able to pass this protection.
But they are "not economically adapted" in my case 😉
Is there a place to post evolution requests and a description of the problem?
Thanks
7 Replies
What proxy you are using? If you blocked from your own IP its expected case and that's why proxies always recommended for scraping
other-emeraldOP•3y ago
Using domestic or residential do not change the problem.
It's works just fine to chain manually 2 URLs with Chrome. (first one is homepage to get reese84 cookie (kind of authorization cookie) and the second one to get datas)
But it does not work with Crawlee or Puppeteer stealth although it works fine with brightdata or zenrows.
=> That's why it may be a fingerprint problem?
BrightData is proxy, so if it works with BD means it works with proxy, right? Puppeteer might be detected and blocked, so try with Playwright, see if targeted page reached
other-emeraldOP•3y ago
As I tried with Crawlee with Playwright, I don't think it will work neither.
national-gold•3y ago
How you use BrightData? Playwright + Firefox + good proxies is usually the beset
other-emeraldOP•3y ago
Thanks Lukas!
I'm testing a new service of BrightData but it's too expensive.
Manually sequencing 2 URLS works with my domestic IP and Chrome but not with Firefox!
national-gold•3y ago
Interesting, can you please paste the flow here as well. @petrpatek. will look as we generally have better experience with Firefox but there can be some fingerprint missmatch