per-site interval between requests?
Imagine the request queue of Crawlee (PlaywrightCrawler) containing URLs of two (or more) sites:
example.com/url1
another-site.com/url2
example.com/url3
another-site.com/url4
...
I would like to configure Crawlee to have per-site interval between requests. For the above example it means:
example.com: 20 sec (or more) between requests
another-site.com: 60 sec (or more) between requests
How to do this with Crawlee?
1 Reply
quickest-silver•3y ago
just add put wait function in
preNavigationHooks