F
Firecrawl•6mo ago
leminh

python sdk crawl endpoint run non stop and run me out of credit

aI was testing firecrawl for the following use case: scrape the website of a startup for their product I notice that with this website in particular, despite setting a limit of 2, the endpoint run non stop until I reach my credit limit app = FirecrawlApp(api_key=[MY_API]) crawl_status = app.crawl_url( 'https://www.oceanfarmr.com/', params={ 'limit': 2, 'scrapeOptions': {'formats': ['markdown', 'html']} }, )
3 Replies
mogery
mogery•6mo ago
hey! new SDK needs the limit and options to be set like so: 
app = FirecrawlApp(api_key=[MY_API])
crawl_status = app.crawl_url(
  'https://www.oceanfarmr.com/', 
  limit=2, 
  scrapeOptions={'formats': ['markdown', 'html']} 
)
app = FirecrawlApp(api_key=[MY_API])
crawl_status = app.crawl_url(
  'https://www.oceanfarmr.com/', 
  limit=2, 
  scrapeOptions={'formats': ['markdown', 'html']} 
)
Tigerr
Tigerr•6mo ago
I was looking at the documentation. it's not clear to me exactly what limit means (is it credits?) Also is there a way of stating depth of the recursive crawling?
mogery
mogery•6mo ago
maxDiscoveryDepth limit is max number of pages/URLs crawled

Did you find this page helpful?