CA
magic-amber
Moving from Playwright to Crawlee/Playwright for Scraping
Are there actually any ressources on building a scraper with crawlee except the one in the docs?
Where do I set all the browser context for example?
3 Replies
Someone will reply to you shortly. In the meantime, this might help:
In the launch context, here's an example https://docs.apify.com/sdk/js/docs/examples/playwright-crawler
Playwright crawler | SDK for JavaScript | Apify Documentation
This example demonstrates how to use PlaywrightCrawler
Or within the pre navigation hook
Something like:
const crawler = new PlaywrightCrawler({
preNavigationHooks: [
async ({ page, request, browserContext }) => {
// Set a specific user agent for the browser context
await browserContext.addCookies([
{ name: 'session', value: '12345', domain: 'example.com' },
]);
// Emulate a specific device (e.g., mobile)
await page.setUserAgent(
'Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1'
);
},
],
requestHandler: async ({ page, request }) => {
console.log(
Visiting ${request.url}
);
const content = await page.content();
console.log(Content length: ${content.length}
);
},
});
await crawler.run(['https://example.com']);