Override default terminal logging
I have a trivial nuance if anyone could point me in the right direction. I have a crawler setup using the CheerioCrawler where I am pulling urls from a database in batches and adding them to the requestQueue.
Everything is working fine however for my specific application I do not want to retry any urls if any type of error occurs (ssl, status codes, etc) mainly due to proxy rotation etc the details are not important.
Which brings me to my problem. Is there any way to override crawlee's default terminal logging as it is clogging up the terminal with errors and stack traces due to reaching maximum retries.
ERROR CheerioCrawler: Request failed and reached maximum retries. RequestError: The proxy server rejected the request with status code 502 (Bad Gateway):
ERROR CheerioCrawler: Request failed and reached maximum retries. RequestError: 18530000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:355:
with long stack traces
when creating the crawler I have the maxRequestRetries set to 0
I have also set the noRetry option to true in each requests constructor
however I am still getting the maximum retries ERROR logged to terminal
Even if I could just turn off the stack traces.
Thanks
1 Reply
Hello @MadFox ,
I am not 100% sure on this one.
Have you tried set custom implementation for
On the
CheerioCrawler constructor?