I'm facing this issue of incomplete crawling where the actual count of pages for the website is different from what the crawl job is able to scrape.
The /map endpoint is giving correct count of URLs in the sitemap, which is 104. [ website - freenome.com ]
but the crawling job is able to scrape sometimes 55, 28 or 29 pages only. Peculiar thing which I have noticed is that the credit usage is also high irrespective of the actual scraped pages.
Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.