CA
Crawlee & Apify•2y ago
correct-apricot

How to persist the context with Crawlee?

Hey, so I am fairly new to Crawlee and have spent the better part of two days trying to figure out how to have persistent context when crawling with Crawlee. I am scraping a website that requires login, and I would like to avoid the overhead of always logging in when a new crawler is ran. I am able to very easily create and persist the context and cookies in the project root directory without Crawlee, only using PlaywrightCrawler like so: 
import { firefox } from 'playwright';
const browser = await firefox.launchPersistentContext('./persistent-context', {
  headless: false,
});
import { firefox } from 'playwright';
const browser = await firefox.launchPersistentContext('./persistent-context', {
  headless: false,
});
can anyone point me in the right direction on how to achieve the same with Crawlee?
1 Reply
Pepa J
Pepa J•2y ago
Hello @robot , You may try persist the cookies to the key-value-store and on start of every necessary handler load them back 
const keyValueStore = await KeyValueStore.open(); // opens default key-value store

// login
await keyValueStore.setValue('cookies', await page.cookies())

// load
await page.cookies(await keyValueStore.getValue('cookies'))
const keyValueStore = await KeyValueStore.open(); // opens default key-value store

// login
await keyValueStore.setValue('cookies', await page.cookies())

// load
await page.cookies(await keyValueStore.getValue('cookies'))

Did you find this page helpful?