Post scrape/crawl big datasets
Hi all, looking for suggestions on how to handle large datasets.
I am writing Jira documentation and I see AI cannot handle reading a whole page when documentation is too big, when fire crawl takes the whole page. Any good practices after I scrape with firecrawl? Like Using qdrant or pinecone but affordable? And best practices? Thanks a lot in advance.
1 Reply
Instead of scraping/crawling and then passing the data into AI to help write your documentation, you might be able to use the Extract endpoint to do both of those steps at the same time.
Here's a link to the playground where you can give it a try: https://www.firecrawl.dev/extract-playground