One-proxy, many-sessions?
Context: Proxy providers often provide a single
proxyUrl from which any number of connections can be opened, i.e. each connection having a different IP despite proxyUrl all being seemingly the same. I wonder, is Crawlee able to create e.g. 100 Sessions (and rotate upon retire/markBad), despite there only being one proxyUrl specified? From reading the docs, a round robin rotation mechanism is referred to and a sessionId-proxyUrl pair is mentioned, so I get the sense that Crawlee every one Session locks exactly one proxyUrl, making redundant the setting sessionPoolOptions.maxPoolSize > 0 . In other words, each proxyConfiguration.proxyUrl can have max. 1 session attached to it.
Hypothesis: When ProxyConfiguration.proxyUrls.length === 1, even though e.g. sessionPoolOptions.maxPoolSize === 100, Crawlee can/will only create one Session because Crawlee thinks there's only one IP available (or something). Does this happen?
Sorry for long winded question, it's my first time using Crawlee and I'm unsure how the details fit together. Thanks for any attention this may get.2 Replies
unwilling-turquoise•2y ago
Hey there! It should still populate the pool with the desired number of sessions, i.e. it will still create more sessions even with one proxyUrl available.
eager-peachOP•2y ago
@Andrey Bykov thank you so much! 🙏
I hope to return the favor by helping others here in the near future, after I complete this project
have a blessed day 🙂