Moinul Moin - asked firecrawl X handler, nobody...
asked firecrawl X handler, nobody answered, so asking here too
for scraping specific category pages from a site, for example, only documentation pages from a documentation site, which one preferable and more accurate,
1. /map+manually url filtering with AI + /scrapebatch
or 2. /crawl with a clear instructions prompt,
i think 1st one
also I found /map is unreliable like sometimes it doesnt get enough links
3 Replies
Hey @Moinul Moin if you want to crawl all the link from the specific site, you should use crawl with excludePaths and includePaths to filter the url results.
I want to crawl all the path comes with the given base url and the include/exclude paths not may help i blv as different site may b different things, i mostly wanted to do work with any documentation website,
thats why thinking map+manual filter+batch scraping
lmk if I am thinking good? @Gaurav Chadha
If the map is working as expected for your URL, then it's good.