/map endpoint seems to have an arbitrary limit of ~300 urls
I've attempted to use the /map endpoint for several domains, but I've struggled to get the full list of urls that exist. Even when I use a search paramater, the limit still seems to be about 300. I've tested this both with the Playground and API.
For example, if I do the /map endpoint for simplicity.com, it responds with ~300 general urls within that overarching domain (even though I know there are 4000+ urls on the site). If I do the /map endpoint with search term '/butterick/' as a subfolder, I'll continue to get ~300 urls that belong within the '/butterick/' subfolder — however, I know for a fact that there are at least 808 urls (products) within that directory.
I've tested with domains including thefoldline.com (302 results) and simplicity.com (308 results) but feel like I'm just burning credits at this point. Given that the domain should be returning A LOT more URLs and the response is consistent regardless of search terms, can someone explain why this is happening?
2 Replies