Crawl results of url/ vs url/index.html are different

I tried the Crawl function on the playground and also my code by using the SDK. I noticed that if I use the url with index.html at the end, it only crawls the single page. However if I remove the index.html, it crawls the page and all the links in the page.

Is this expected? Why they are handled differently? Thanks!
Screenshot_2025-01-09_at_11.38.23_PM.png
Screenshot_2025-01-09_at_11.38.39_PM.png
Was this page helpful?