Firecrawl•14mo ago

Issue with Crawling Multiple Pages - Only One Page is Crawled

I am experiencing an issue where I am unable to crawl multiple pages; only a single page is being crawled. What is happening: I am trying to crawl multiple pages, but the crawler is only able to crawl a single page. Environment: I am self-hosting using Docker. Setup: I followed the official SELF_HOST.md guide for the setup. Could you please assist me in resolving this issue?

5 Replies

testusukeOP•14mo ago

I tried to crawl below site. https://www.promptingguide.ai/

Prompt Engineering Guide – Nextra

A Comprehensive Overview of Prompt Engineering

testusukeOP•14mo ago

Additionally, I was able to successfully crawl multiple pages without any issues using the SaaS version of FireCrawl.

Adobe.Flash•14mo ago

Hey @testusuke could you try pulling the repo again? We made an update last night that probably solves your issue! I tested locally and seems to be working

testusukeOP•14mo ago

Thank you for your quick response. I confirmed that everything is working!

Benos•14mo ago

I am having a similar issue on self hosted, trying to get Python documentation (only 3.11). It is only returning the first page but the list of links include a bunch of valid ones to crawl. I just pulled the repo on main and still having the issue

crawl_status = app.crawl_url(
  'https://docs.python.org/3.11/',
  params={
      'limit': 50,
      'maxDepth': 10,
      'ignoreSitemap': True,
        'includePaths': ["/3.11/*"],
          'scrapeOptions': {
              'formats': ['markdown', 'links'],
                'includeTags': ['.body']
          }
      }
)

crawl_status = app.crawl_url(
  'https://docs.python.org/3.11/',
  params={
      'limit': 50,
      'maxDepth': 10,
      'ignoreSitemap': True,
        'includePaths': ["/3.11/*"],
          'scrapeOptions': {
              'formats': ['markdown', 'links'],
                'includeTags': ['.body']
          }
      }
)

Gaming

Programming

Issue with Crawling Multiple Pages - Only One Page is Crawled

Did you find this page helpful?