config logs: 1/ add timestamp 2/save to file

Hello Now logs from Crawlee looks like:

INFO  PuppeteerCrawler: Starting the crawl
DEBUG PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to ...
DEBUG PuppeteerCrawler:SessionPool: Created new Session - session_fLcLuWwBqm
DEBUG PuppeteerCrawler:SessionPool: Created new Session - session_ufpfTcwERo

Questions: 1. How to add timestamp to logs? Like:

2023-01-25 12:34:58.123 INFO  PuppeteerCrawler: Starting the crawl
2023-01-25 12:34:58.456 DEBUG PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to ...

2. How to save logs in file [and possibly also watch on console]? BTW. should I put question here or better on stack-over-flow and push link here? Thanks and cheers GT

3 Replies

extended-yellow•3y ago

To add timestamps to your logs, you can override the default logger settings:

import {
    BasicCrawler,
    log,
} from 'crawlee';

log.options.logger.setOptions({
    skipTime: false,
});

const basic_crawler = new BasicCrawler({ ... });

import {
    BasicCrawler,
    log,
} from 'crawlee';

log.options.logger.setOptions({
    skipTime: false,
});

const basic_crawler = new BasicCrawler({ ... });

To save logs in file [and possibly also watch on console], you can customize your logging further by replacing the default logger instances with your own implementations.

wise-whiteOP•3y ago

It works, great, thanks! ;] BTW. Where can I find options like skipTime? In very short, rest I will find by myself. Tnx!

extended-yellow•3y ago

found here : https://github.com/apify/apify-shared-js/blob/master/packages/log/src/log.ts#L107

GitHub

apify-shared-js/log.ts at master · apify/apify-shared-js

Utilities and constants shared across Apify projects. - apify-shared-js/log.ts at master · apify/apify-shared-js

Gaming

Programming

config logs: 1/ add timestamp 2/save to file

Did you find this page helpful?