goodwood - Which endpoint should I use to retur...
Which endpoint should I use to return all open job opportunities, in a structured output, from a list of company websites? Scrape, Crawl, Extract, or a mix?
6 Replies
Hi @goodwood you should try /search - https://docs.firecrawl.dev/features/search it should fit best for your usecase and for structured use JSON format.
Firecrawl Docs
Advanced Scraping Guide | Firecrawl
Learn how to improve your Firecrawl scraping with advanced options.
Hi @Gaurav Chadha, thanks for the reply. Correct me if I'm wrong but I don't think searching the web is what I'm looking for. I already have a specific list of companies and their domains that I want to scrape.
I'm just not clear on which endpoint is best for this. It seems that they all can return structured output which is what we want.
Okay, if you've multiple site URLs, then you can use extract, it should work best.https://docs.firecrawl.dev/features/extract
Let me add more context. I'm @goodwood's coworker and an engineer. We have a list of URLs to company career pages, ex. https://jobs.silkroad.com/Numotion/Careers. The goal is to use Firecrawl in the most efficient/resilient way possible to go to each URL, potentially filter the jobs down a bit (either in through the URL or filters present on the page), and get all the information on each resulting job.
Thanks for the info. The Extract API is the best for this use case.
OK we'll give it a shot, thank you! 🚀