Extract Product URL's
I'm trying to extract all the product URL's from a page
eg. https://www.james-porter.co.uk/watches/all-watches OR https://watchesofknightsbridge.com
The issue is that some websites have pagination, and some don't.
I also tried to map the whole website, but the issue is that some of them have different type of product url's than other
eg website.com/3232
OR
website.com/product/Black-golden-watch
OR
website.com/watches/vintage/Vintagerolex
I figured out how to extract the information that I want once I get the exact product page, but I still don't know how to extract the exact product page every time on different websited.
Curated
Home - Curated
A curated selection of the most sought after vintage and modern watches for direct sale.
1 Reply
@VasD it sounds like you might want to use the Map or Crawl endpoints for this? For example, with the Map endpoint you can cheaply get the list of URLs: https://www.firecrawl.dev/app/playground?url=https%3A%2F%2Fwww.james-porter.co.uk%2Fwatches%2Fall-watches&mode=map&limit=10&excludes=&includes=&formats=markdown&onlyMainContent=true&excludeTags=&includeTags=&proxy=undefined&includeSubdomains=true&mapSearch=&uniqueKey=1748046094835