FirecrawlF
Firecrawl•17mo ago
Julia from Storia

Firecrawl doesn't seem to crawl everything

I'm trying to run Firecrawl on pytorch's documentation, and I merely get ~15 results with these URLs:
https://pytorch.org/docs/stable
https://pytorch.org/docs/stable/distributed.pipelining.html
https://pytorch.org/docs/stable/dynamo/index.html
https://pytorch.org/docs/stable/fsdp.html
https://pytorch.org/docs/stable/fx.html
https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html
https://pytorch.org/docs/stable/generated/torch.set_num_threads.html
https://pytorch.org/docs/stable/hub.html
https://pytorch.org/docs/stable/index.html
https://pytorch.org/docs/stable/jit.html
https://pytorch.org/docs/stable/library.html
https://pytorch.org/docs/stable/torch.compiler.html
https://pytorch.org/docs/stable/torch.compiler_aot_inductor.html
https://pytorch.org/docs/stable/torch.compiler_get_started.html
https://pytorch.org/docs/stable/torch.compiler_troubleshooting.html

Clearly it's missing out on a whole lot of pages. Here's how I'm calling it:
crawl_result = firecrawl_app.crawl_url(
    "https://pytorch.org/docs/stable",
    params={"crawlerOptions": {"maxDepth": 10_000, "limit": 10_000}}
)

Am I missing something? Are any of the other default parameters truncating the crawl somehow?
Was this page helpful?