FirecrawlF
Firecrawl•10mo ago
SunburntRock89

crawling only main content returns content from nav tags

Hi there,
I'm crawling with these options
{
    "url": "https://developer.apple.com/design/human-interface-guidelines/foundations",
    "maxDepth": 2,
    "scrapeOptions": {
        "formats": [
            "html"
        ],
        "onlyMainContent": true
    },
    "deduplicateSimilarURLs": true,
    "allowBackwardLinks": true,
    "allowExternalLinks": false,
    "ignoreQueryParameters": true,
    "limit": 15
}


This is returning links like 
'https://developer.apple.com/news/'
'https://developer.apple.com/discover/'
'https://developer.apple.com/develop/'
'https://developer.apple.com/distribute/'
'https://developer.apple.com/design/'


These are part of the header of the page, and I can't seem to get firecrawl to ignore them and only send me stuff from the main areas
Was this page helpful?