Bulk Crawling
Hello,
In my app, users enter a URL and select additional links from a list returned by the /map endpoint that they would like to crawl alongside the main URL they provided. Currently, I'm sending a request to the /crawl endpoint with the main URL being the one the user provided, and the includePaths:[] containing the paths the user selected from the /map endpoint.
However, it seems that all the links from the main URL are being crawled, in addition to the selected paths. How can I modify my request so that only the main URL and its selected includePaths are crawled, excluding any other links?
In my app, users enter a URL and select additional links from a list returned by the /map endpoint that they would like to crawl alongside the main URL they provided. Currently, I'm sending a request to the /crawl endpoint with the main URL being the one the user provided, and the includePaths:[] containing the paths the user selected from the /map endpoint.
However, it seems that all the links from the main URL are being crawled, in addition to the selected paths. How can I modify my request so that only the main URL and its selected includePaths are crawled, excluding any other links?
