Help Request: /map + search not returning product URLs
I’m using the /v2/map incl. search query endpoint to discover product detail page URLs, but I’ve run into an issue where some PDPs aren’t being returned even though they’re live and accessible on the site.
Example
Site: https://www.bstgroup.eu
Query: "PRF Mouse"
PDP URL I expect:
https://www.bstgroup.eu/producten/zakelijke-oplossingen/sector/onderwijs/bakkerelkhuizen-prf-mouse-wireless/
Current call: url = "https://api.firecrawl.dev/v2/map" payload = { "url": "www.bstgroup.eu", "search": "BakkerElkhuizen PRF", "limit": 5000, "includeSubdomains": True, "sitemap": "include" }The response only gives me higher-level category/listing pages, but not the actual PDP (linked above). I’ve tried using different search strings ("PRF Mouse Wireless", "BakkerElkhuizen PRF") but still the PDP isn’t included. Questions Is this expected behavior of /map? Any recommended best practices for reliably discovering PDP URLs?
1 Reply
@Sam
/map is designed to scrape pages of a website related to a specific topic, when the user only needs to scrape specific pages of a website, whereas crawl scrapes all subpages as well.
since onderwijs/bakkerelkhuizen-prf-mouse-wireless/ is a subpage, I'll recommend you to use /crawl