externalUrls
Does anyone know definitively how the allowExternalLinks=true behaves? I was looking at the code and it seems it will only go a single page deep into an external domain, but when doing a /crawl and enabling that option, it seems like we are getting huge explosion of pages, which seems relatively unbounded.
1 Reply
@a6coin I'd recommend using the
maxDiscoveryDepth to control how deep you crawl. This will help keep the huge explosion of pages under control.