Ash Barbour - Max depth is not working - I set ...
Max depth is not working - I set it to 1 and still I get URLs with like 3 or 4 page depth
Did I miss something?
1 Reply
Hi @Ash Barbour, actually it's correct.
maxDiscoveryDepth: Controls how many link-following hops the crawler makes (0 = just the starting URL, 1 = starting URL + all URLs it links to, 2 = those + all URLs they link to, etc.).
So for lemonade.com what's actually happening is:
- Discovery depth 0: The root URL https://lemonade.com is crawled
- Discovery depth 1: All URLs linked from that root page are discovered and crawled
- Discovery stops: No further link discovery happens from those depth-1 pages
This means if lemonade.com homepage links to (example) https://www.lemonade.com/homeowners/explained/what-homeowners-insurance-covers/ (a 4-level deep URL), that URL will be crawled because it was discovered at depth 1 (one hop from the root).
If you want to filter the URLs, you can use includePaths with a regex pattern example:
Hope this clear what is happening with maxDepth.Insurance Built For the 21st Century | Lemonade
Lemonade, America’s top-rated insurance company, protects your family and your belongings—at home, and everywhere else. Sign up in seconds, get paid in minutes.
Lemonade
What Does Homeowners Insurance Cover (And What's Not Covered)?
When you think ‘homeowners insurance,’ easy-to-read probably doesn’t come to mind. Here are your biggest insurance Qs, answered.