Ideas for increasing speed? Fast mode?
What are some ways to increase crawl speed? How does fast mode work?
3 Replies
Hey Druvs.
Speed is one of our core objectives right now, only falling second to reliability. We actually just shaved off 900ms from every scrape today (which should increase the speed of crawl as well)
In the meantime, i'd suggest setting max depths / exclude paths if you can to improve the speed. however, the tradeoff here is that you will not get as much data back!
@Adobe.Flash Is fastMode still something you'd reccomend?
Yep, Fast mode should increase by 4-5x for websites that don't have a sitemap. Although it might compromise on quaility on some js-heavy websites, it should work pretty well.
Understood! Is the crawl request parallelised so more crawlers can work concurrently?
How exactly does the fast mode work? Assuming there is some trade-off? @Adobe.Flash