How to delay
When scraping a particular news site, I would like to include a delay to prevent the site from becoming too busy. Is such an option already provided? If not, please let me know what part of the code I should use to sleep.
1 Reply
passive-yellow•2y ago
you can put sleep function in the requestHandler function you can also reduce maxConcurrency of the crawler