Include Only Paths and Exclude Paths not working as intended
Hi guys, I am using the Playground to do a simple Crawl on a single page and only get links with /buy/* , but the Include Only Path and Exclude does not work and filter down the links returned
Am I missing something?
crawl_result = app.crawl_url('https://www.propertyfinder.eg/en/search?c=1&fu=0&ob=mr', params={
'limit': 1,
'maxDepth': 0,
'excludePaths': [ '(/en/search*|rent/.+|en/search?.+)' ],
'includePaths': [ 'buy/.+' ],
'scrapeOptions': {
'formats': [ 'markdown', 'links' ],
'onlyMainContent': False
}
})`
response keeps returning irrelevant links
I tried many ways to do the include and exclude
Am I missing something?
crawl_result = app.crawl_url('https://www.propertyfinder.eg/en/search?c=1&fu=0&ob=mr', params={
'limit': 1,
'maxDepth': 0,
'excludePaths': [ '(/en/search*|rent/.+|en/search?.+)' ],
'includePaths': [ 'buy/.+' ],
'scrapeOptions': {
'formats': [ 'markdown', 'links' ],
'onlyMainContent': False
}
})`
response keeps returning irrelevant links
I tried many ways to do the include and exclude
Property Finder
