Crawl results of url/ vs url/index.html are different
I tried the Crawl function on the playground and also my code by using the SDK. I noticed that if I use the url with index.html at the end, it only crawls the single page. However if I remove the index.html, it crawls the page and all the links in the page.
Is this expected? Why they are handled differently? Thanks!
Is this expected? Why they are handled differently? Thanks!
