External Webpage Scraping
Hello All,
I have a question: If I scrape a website and it contains links to other external websites, will those linked pages also be scraped automatically?"
1 Reply
Great question! It sounds like you might be referring to the Crawl endpoint, not the Scrape endpoint. Scrape just scrapes a single page.
By default, it won't crawl onto other websites, however, you can control this with allowExternalLinks (defaults to false) . See https://docs.firecrawl.dev/api-reference/endpoint/crawl-post#body-allow-external-links.