Need Scraping Benchmarking details
I have a use case that involves scraping ~ 10,000 products at a given time. It would be helpful to understand any available data or insights regarding:
1. Latency per request or per batch. I understand FireCrawl has built-in batching logic, overall latency would help.
2. Accuracy of extracted data (especially for structured fields like price, title, description, etc.)
3. Reliability and consistency at scale
4. Credits used per page scrape. The doc says 1 credit/page but actually it is consuming 5 credits per page scrape.
5.Timeline regarding Search functionality in Dify node
6. Can I combine both search & scrape in one single call? also have a custom logic to scrape only a few top links per site?
1 Reply
We've done some internal benchmarks and hoping to share some of those out soon!
Basic scrapes cost 1 credit yet, but it can cost more for PDF scraping or advanced features (stealth mode, JSON mode). These are all mentioned at https://www.firecrawl.dev/pricing.
Yes, /search is designed to search and scrape at the same time. But you can't define custom logic to scrape only a few top links per site.