Bulk Crawling
Hello,
In my app, users enter a URL and select additional links from a list returned by the /map endpoint that they would like to crawl alongside the main URL they provided. Currently, I'm sending a request to the /crawl endpoint with the main URL being the one the user provided, and the includePaths:[] containing the paths the user selected from the /map endpoint.
However, it seems that all the links from the main URL are being crawled, in addition to the selected paths. How can I modify my request so that only the main URL and its selected includePaths are crawled, excluding any other links?
6 Replies
Hi there, would you only like to get the content from the selected paths? Or would you like to further crawl on them as well
Only the the content from the selected paths
Then you'll probably want to send multiple
/scrape requests insteadWhat is the rate limit for /scrape ?
It depends on your pricing tier -- https://www.firecrawl.dev/pricing
We're considering adding a bulk scrape endpoint for this use case