F
Firecrawl•7mo ago
SunburntRock89

crawling only main content returns content from nav tags

Hi there, I'm crawling with these options 
{
    "url": "https://developer.apple.com/design/human-interface-guidelines/foundations",
    "maxDepth": 2,
    "scrapeOptions": {
        "formats": [
            "html"
        ],
        "onlyMainContent": true
    },
    "deduplicateSimilarURLs": true,
    "allowBackwardLinks": true,
    "allowExternalLinks": false,
    "ignoreQueryParameters": true,
    "limit": 15
}
{
    "url": "https://developer.apple.com/design/human-interface-guidelines/foundations",
    "maxDepth": 2,
    "scrapeOptions": {
        "formats": [
            "html"
        ],
        "onlyMainContent": true
    },
    "deduplicateSimilarURLs": true,
    "allowBackwardLinks": true,
    "allowExternalLinks": false,
    "ignoreQueryParameters": true,
    "limit": 15
}
This is returning links like 
'https://developer.apple.com/news/'
'https://developer.apple.com/discover/'
'https://developer.apple.com/develop/'
'https://developer.apple.com/distribute/'
'https://developer.apple.com/design/'
'https://developer.apple.com/news/'
'https://developer.apple.com/discover/'
'https://developer.apple.com/develop/'
'https://developer.apple.com/distribute/'
'https://developer.apple.com/design/'
These are part of the header of the page, and I can't seem to get firecrawl to ignore them and only send me stuff from the main areas
0 Replies
No replies yetBe the first to reply to this messageJoin

Did you find this page helpful?