Crawlee & Apify•2y ago

chromium.launchpersistentcontext with crawlee

Hi everyone, this doc: https://docs.apify.com/academy/puppeteer-playwright/browser-contexts shows how to use persistent context when working with pure playwright. But how can I combine this with crawlee? is there a configuration for this while calling PlayWrightCrawler(...)? or a way to get similar behaviour?

VI - Creating multiple browser contexts | Academy | Apify Documenta...

Learn what a browser context is, how to create one, how to emulate devices, and how to use browser contexts to automate multiple sessions at one time.

4 Replies

Pepa J•2y ago

Hi @levav , The persistentContext is just a tool, it depends for which purpose you want to use it, there might be better solution for your needs. Can you describe little bit more what are you trying to achieve?

diverse-roseOP•2y ago

Hi @Pepa J thank you for responding. I'm trying to essentially create a pool of logged in users for instagram scrapping. I have a script that creates a user, a script that logs in, and I want to maintain a pool of browser contexts that are saved and loaded from some file when I rerun the script, so that the script will only log in if a cookie expired & only create an account when one was blocked.

Pepa J•2y ago

Thanks for the description. So the flow could be: - check KeyValueStore for valid cookies - if there are any check if they are not expired and use them - otherwise do the login again and save new cookies to the KeyValue Store. - I also suggest to use

launchContext: {
    useIncognitoPages: true
},
browserPoolOptions: {
  maxOpenPagesPerBrowser: 1
}

launchContext: {
    useIncognitoPages: true
},
browserPoolOptions: {
  maxOpenPagesPerBrowser: 1
}

Another approache would be to use BasicCrawler and managing Playwright contexts totally by yoursel inside the handlers.

diverse-roseOP•2y ago

Thank you! I'll try doing that 🙂

Gaming

Programming

chromium.launchpersistentcontext with crawlee

Did you find this page helpful?