External Webpage Scraping

Hello All,
I have a question: If I scrape a website and it contains links to other external websites, will those linked pages also be scraped automatically?"

micah.stairs•5/25/25, 12:50 AM

Great question! It sounds like you might be referring to the Crawl endpoint, not the Scrape endpoint. Scrape just scrapes a single page.

By default, it won't crawl onto other websites, however, you can control this with allowExternalLinks (defaults to false) . See https://docs.firecrawl.dev/api-reference/endpoint/crawl-post#body-allow-external-links.

External Webpage Scraping

Similar Threads

Similar Threads

Similar Threads