CA
Crawlee & Apify•3mo ago
other-emerald

Loadtime

Hello, Is there a way to get the load time of a site from crawlee in headless mode? I'm using PlaywrightCrawler. Thanks!
3 Replies
Hall
Hall•3mo ago
Someone will reply to you shortly. In the meantime, this might help:
Pepa J
Pepa J•3mo ago
Hi @BOBPG : Do you have something like this in your mind? 
const crawler = new PlaywrightCrawler({
    // ...
    preNavigationHooks: [async ({ request }, gotoOptions) => {
        requestStartTimes.set(request.id!, Date.now());
        gotoOptions!.waitUntil = 'load';
    }],
    requestHandler: async ({ request, log }) => {
        const loadingTime = Date.now() - requestStartTimes.get(request.id!)!;
        log.info(`Loading ${request.url} took ${loadingTime / 1000}s`);
    },
});
const crawler = new PlaywrightCrawler({
    // ...
    preNavigationHooks: [async ({ request }, gotoOptions) => {
        requestStartTimes.set(request.id!, Date.now());
        gotoOptions!.waitUntil = 'load';
    }],
    requestHandler: async ({ request, log }) => {
        const loadingTime = Date.now() - requestStartTimes.get(request.id!)!;
        log.info(`Loading ${request.url} took ${loadingTime / 1000}s`);
    },
});
other-emerald
other-emeraldOP•3mo ago
Someting like this looks great, i'm going to try this, thanks!

Did you find this page helpful?