linksOnPage
Was wondering for the crawl API , did the response format change i.e. before I remember it would showcase all the urls crawled from a page + markdown content in seperate JSON objects, but now it doesn't and instead I get linksOnPage.
def crawl_ir_url(url: str) -> List[Dict[str, Any]]:
print(f"Initiating the crawl on Firecrawl for URL: {url}")
crawl_json = app.crawl_url(url, params={
'crawlerOptions': {
'limit': 200
},
'pageOptions': {
'onlyMainContent': True,
'screenshot': True,
'waitFor': 10000,
}
}, wait_until_done=True)
Is there a way to configure the old behaviour or is it now the case in order to crawl a URL, need to specify to crawl the linksOnPage?
def crawl_ir_url(url: str) -> List[Dict[str, Any]]:
print(f"Initiating the crawl on Firecrawl for URL: {url}")
crawl_json = app.crawl_url(url, params={
'crawlerOptions': {
'limit': 200
},
'pageOptions': {
'onlyMainContent': True,
'screenshot': True,
'waitFor': 10000,
}
}, wait_until_done=True)
Is there a way to configure the old behaviour or is it now the case in order to crawl a URL, need to specify to crawl the linksOnPage?
