PuppeteerCrawler proxy rotate
I'm using PuppeteerCrawler in Crawlee lib. I want to rotate proxy, how to apply proxy rotate to PuppeteerCrawler?
Thanks!
4 Replies
ambitious-aqua•3y ago
Heya! Check out the
0:36 mark of this video to see how to rotate proxies in Crawlee using ProxyConfiguration:
https://www.youtube.com/watch?v=g1Ll9OlFwEQ&t=36s
It is done automatically when you use this constructor.Apify
YouTube
Crawlee, the web scraping and browser automation library
Visit https://crawlee.dev/ to get started.
Crawlee helps you build reliable scrapers. Fast. It is an intuitive, customizable open-source Node.js library for web scraping and browser automation.
Quickly scrape data, store it, and avoid getting blocked with headless browsers, smart proxy rotation, and auto-generated human-like headers and finger...
fair-roseOP•3y ago
I don't want to auto. I want to control from my UI admin to config
ambitious-aqua•3y ago
You want to control which proxies are used, or when they are rotated?
You can manually control when a proxy is rotated with
session.retire() and some config in SessionPoolOptionsambitious-aqua•3y ago
Session | API | Crawlee
Sessions are used to store information such as cookies and can be used for generating fingerprints and proxy sessions.
You can imagine each session as a specific user, with its own cookies, IP (via proxy) and potentially a unique browser fingerprint.
Session internal state can be enriched with custom user data for example some authorization toke...