How to set concurrency for Firecrawl self-hosted
I'm setting max_concurrency in my crawl options but doesn't seem to do anything. the docker service is still running scrapers serially.
1 Reply
Hi @rno321 In a self-hosting setup, you'll need to configure the
docker-compose.yaml to set the max_concurrency and provide a larger number of workers to make the concurrency work. I've added a PR that you can refer to for setup - https://github.com/firecrawl/firecrawl/pull/2193GitHub
feat: [self-host] - add support to configure concurrency for crawl ...
tested ok:
Summary by cubic
Adds configurable concurrency controls for self-hosted crawling to tune throughput and resource usage. Defaults are provided and mapped to the Playwright service with...