crawling isn't getting accurate data from site
when i am trying to crawl through my site using the playground, I am unable to retrieve all the numerical data from the site. and extract feature is also returning different results on re run even with similar configuration and prompt.
link : https://softr.biglook.ai/
Inventory Management
Dashboard
Made with Softr, the easiest way to turn your data into portals and internal tools.
3 Replies
Without more info about the URL you are trying to crawl and your request parameters, it's going to be hard for anyone to give you guidance.
my bad, i have added the link, this is just a demo website we had created to test firecrawl
Here are my favorite parameters to try when I'm having trouble scraping a particular site:
- proxy="auto": This falls back to the more expensive but more reliable stealth mode if the basic scrape didn't work
- waitFor=3000: This gives the website some extra time for the content to load before we scrape its content
- onlyMainContent=false: The logic to identify the main content doesn't work as expected on certain sites
- timeout=60000: A timeout of 1 minute gives the scrape a bit more buffer room, just in case