Maintain the same browser/scope
Hi!
I'm having a issue while scraping a web app. This app have heavy use on context and cookies, and when I enqueue over 80 urls using EnqueueLink, after about the 20th url scraped, my algorithm opens another browser windows loosing scope, and loosing access to the urls which needs the context or cookes.
So, is there any way to config Crawlee to avoid opening more browsers? Or maybe if is there any way to keep the first scope even between browsers.
Thanks!
4 Replies
conscious-sapphire•3y ago
Hey there!
The main thing is that by default crawlee allows 20 open tabs per browser, so you could start by increasing this number: https://crawlee.dev/api/browser-pool/interface/BrowserPoolOptions#maxOpenPagesPerBrowser
Besides that, check also the following options:
1) You could set
retireBrowserAfterPageCount to some high number (100 by default): https://crawlee.dev/api/browser-pool/interface/BrowserPoolOptions#retireBrowserAfterPageCount
2) You could set maxPoolSize of session pool to 1, to make sure you're having one session: https://crawlee.dev/api/core/interface/SessionPoolOptions#maxPoolSize
3) also check the related session options: https://crawlee.dev/api/core/interface/SessionOptionsvicious-goldOP•3y ago
Hi @Andrey Bykov
Thanks for your help, bun unfortunately it didn' work.
I tried with this configuration, but the results are kinda the same. While it resists open more tabs, it opens new browsers anyway.
Do you have any other advice?
conscious-sapphire•3y ago
could you send some runId? but generally - I'd recommend trying to remove minConcurrency and let it on autoscaled pool. Also will try to lower the actor memory. But that's just random thoughts. Please send the runId if possible.
passive-yellow•2y ago
Hi! I have the same issue. Did you manage to solve the problem? Thanks!