excluding file type

Hey everyone, Can the excludes option be used to exclude file types like pdfs?

"crawlerOptions": {
    "excludes":["*.pdf"],
    "returnOnlyUrls": False,
    "maxDepth": 3,
    "mode": "default",
    "limit": 10
},

"crawlerOptions": {
    "excludes":["*.pdf"],
    "returnOnlyUrls": False,
    "maxDepth": 3,
    "mode": "default",
    "limit": 10
},

Whenever I run this I seem to get more "Crawl job failed or was stopped. Status: stuck" errors

2 Replies

Adobe.Flash•16mo ago

Not currently, but that's a very good suggestion! I am adding that as an issue in our GitHub

Adobe.Flash•16mo ago

https://github.com/mendableai/firecrawl/issues/309

GitHub

feat: excludes to be used to exclude file types like pdfs · Issue #...

Ability to use the excludes param to exclude file types in urls: "crawlerOptions": { "excludes":["*.pdf"], "returnOnlyUrls": False, "maxDepth": 3, ...

Gaming

Programming

excluding file type

Did you find this page helpful?