Firecrawl doesn't seem to crawl everything
I'm trying to run Firecrawl on pytorch's documentation, and I merely get ~15 results with these URLs:
Clearly it's missing out on a whole lot of pages. Here's how I'm calling it:
Am I missing something? Are any of the other default parameters truncating the crawl somehow?
https://pytorch.org/docs/stable
https://pytorch.org/docs/stable/distributed.pipelining.html
https://pytorch.org/docs/stable/dynamo/index.html
https://pytorch.org/docs/stable/fsdp.html
https://pytorch.org/docs/stable/fx.html
https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html
https://pytorch.org/docs/stable/generated/torch.set_num_threads.html
https://pytorch.org/docs/stable/hub.html
https://pytorch.org/docs/stable/index.html
https://pytorch.org/docs/stable/jit.html
https://pytorch.org/docs/stable/library.html
https://pytorch.org/docs/stable/torch.compiler.html
https://pytorch.org/docs/stable/torch.compiler_aot_inductor.html
https://pytorch.org/docs/stable/torch.compiler_get_started.html
https://pytorch.org/docs/stable/torch.compiler_troubleshooting.htmlhttps://pytorch.org/docs/stable
https://pytorch.org/docs/stable/distributed.pipelining.html
https://pytorch.org/docs/stable/dynamo/index.html
https://pytorch.org/docs/stable/fsdp.html
https://pytorch.org/docs/stable/fx.html
https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html
https://pytorch.org/docs/stable/generated/torch.set_num_threads.html
https://pytorch.org/docs/stable/hub.html
https://pytorch.org/docs/stable/index.html
https://pytorch.org/docs/stable/jit.html
https://pytorch.org/docs/stable/library.html
https://pytorch.org/docs/stable/torch.compiler.html
https://pytorch.org/docs/stable/torch.compiler_aot_inductor.html
https://pytorch.org/docs/stable/torch.compiler_get_started.html
https://pytorch.org/docs/stable/torch.compiler_troubleshooting.htmlClearly it's missing out on a whole lot of pages. Here's how I'm calling it:
crawl_result = firecrawl_app.crawl_url(
"https://pytorch.org/docs/stable",
params={"crawlerOptions": {"maxDepth": 10_000, "limit": 10_000}}
)crawl_result = firecrawl_app.crawl_url(
"https://pytorch.org/docs/stable",
params={"crawlerOptions": {"maxDepth": 10_000, "limit": 10_000}}
)Am I missing something? Are any of the other default parameters truncating the crawl somehow?
