Crawling inconsistency
Hi, I'm trying to figure out how i can get more consistency with crawling. For example if I run the same crawl job back to back with maxdepth 1 and maxpages 50, the first crawl will have 48 pages crawled, the second 52, and then a few of the pages are different in which are scraped. I know this is likely due to the concurrent scrapes, but is there some way to get more consistency / deterministic results? Let me know
