Crawling website to get all data including pagination data
So there is this website, which has company data and which has pagination. initially it displays data for page 1 which lists 15 company names .I used " https://api.firecrawl.dev/v1/scrape " endpoint with prompt to get all company names, it does only give me 15 company names, which are listed in page 1. How to go about fetching compnay names of those page 2,page 3.... .
Also tried using "https://api.firecrawl.dev/v1/crawl" endpoint. The response is passed on to my webhook url and i do receive data but it is complete page data. I want only the company name data.
This endpoint also does not support prompt, using which i could have got only company names.
Please suggest me how to approach such problem.
Also tried using "https://api.firecrawl.dev/v1/crawl" endpoint. The response is passed on to my webhook url and i do receive data but it is complete page data. I want only the company name data.
This endpoint also does not support prompt, using which i could have got only company names.
Please suggest me how to approach such problem.
