Crawlee Playwright is detected as bot
Checking on this page, Crawlee Playwright is detected as bot due to CDP.
https://www.browserscan.net/bot-detection
This is a known issue, also discussed on:
https://github.com/berstend/puppeteer-extra/issues/899
Wondering if Crawlee can come up with a solution?
BrowserScan
BrowserScan - Robot Detection/WebDriver
Bot Test, WebDriver Test, Discord bots, Cloudflare Turnstile, Google reCAPTCHA, gives you a powerful tool to prevent online fraud
GitHub
[Bug] Stealth being detected by Chrome DevTools Protocol (CDP) · Is...
Puppeeteer stealth is now being easily detected, checkout https://deviceandbrowserinfo.com/learning_zone/articles/detecting-headless-chrome-puppeteer-2024
16 Replies
@Jeno just advanced to level 2! Thanks for your contributions! 🎉
View post on community site
This post has been pushed to the community knowledgebase. Any replies in this thread will be synced to the community site.
Apify Community
environmental-rose•8mo ago
checking with team and getting back to you.
fair-rose•8mo ago
Hi @Jeno
We are working on solution that will not use Playwright and should be more unblockable.
Meanwhile you can checkout these tips https://docs.apify.com/academy/anti-scraping#quick-start
Anti-scraping protections | Academy | Apify Documentation
Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues.
deep-jadeOP•8mo ago
That's exciting news!
sensitive-blue•8mo ago
Very interesting, @Jeno pls look at this screenshot.
As far as i understand: "Normal" = "Nothing detected", right?
Code: Playwright+Crawlee+Firefox+rotating proxies.
And exactly same program is detected on another site, see here:
https://discord.com/channels/801163717915574323/1296398744870457394
deep-jadeOP•7mo ago
Can you give a hint at what stage that new solution is? Weeks or months away? Or next major version?
like-gold•6mo ago
I wanna know that too. It is a great feature .👏
👀
fascinating-indigo•6mo ago
👀
like-gold•6mo ago
👀
flat-fuchsia•6mo ago
GitHub
GitHub - apify/fingerprint-suite: Browser fingerprinting tools for ...
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify. - apify/fingerprint-suite
deep-jadeOP•4mo ago
It's a lot of things. I have read somewhere that CF can detect the synthetic mouse actions by Playwright and Puppeteer. The only solution I had success with was Puppeteer Real Browser. It passes CF easily.
like-gold•4mo ago
How can we use real browsers at scale?
deep-jadeOP•3mo ago
Crawlee silently added Camoufox and it's amazing! Real browser was a pain to use. I am using Camoufox with simple datacenter proxies and easily passing everything. It is available as a template. Make sure you use socks5 to avoid some edge cases.
environmental-rose•3mo ago
haha its still in beta 🙂 we are about to post it on social media this week! glad you liked it 😄