I'm trying to scrape the products that exist on catalog page, and to do so I'm setting up a crawl where I set an Include Only Paths (includesPath), however the crawl only returns the original catalog URL.
In this case I expect to get back all 3 pages back, but I only get the catalog page. I've even tried Allowing backwards links. Is this a bug or am I missing something?
Join builders, developers, users and turn any website into LLM-ready data, enabling developers to power their AI applications with clean, structured information crawled from the web.