SpooKyHammer - Hello, what's the recommend spec...
Hello, what's the recommend spec to self host firecrawl? I am currently trying it out 2 vCPU and 4GB ram but its crashes the server after running successful crawl on 20-30 websites. and does it internally cache things if so how can I disable it?
7 Replies
Hi @SpooKyHammer Your 2 vCPU and 4GB RAM setup is likely hitting, what is the error on logs?
Recommended Steup:
- 4+ vCPU (Firecrawl runs multiple worker processes: API server, queue-worker, 5x nuq-workers, nuq-prefetch-worker, index-worker)
- 8+ GB RAM (to handle browser automation via Playwright, Redis, PostgreSQL, and concurrent scraping jobs)
From logs I can see OOM killed, but what's interesting is it was able to work for first 20-30 websites and then fail rather than fail at the beginning itself
Seems a memory leak issue to me, Try to set NUQ_WORKER_COUNT=2 in your .env to reduce parallel workers from 5 to 2. This significantly reduces memory footprint
Guess so, I saw the CPU Utilisation spike up to 95% with my existing setup is that normal?
yes, because you have very less resources 2 vCPU and 4GB
will that change if I reduce the number of nuq worker to 2 as you suggested earlier?
and honestly I upgraded my ec2 to 2 vCPU and 8GB ram yet I can see it's went up to 87% cpu utilisation today and yesterday it was 95%
I am trying to understand why its using more cpu despite having good amount of ram left approx additional 4GB ram are always free when it's working on the api calls.
Today a couple of Crawl API calls returned 502 errors, the same requests succeeded yesterday.
could you check your check container resource usage in real-time and share container logs?
Yeah, that should reduce the load. The high CPU usage (87-95%) is expected with only 2 vCPUs running 8+ processes. If you could upgrade to 4 vCPUs, that should reduce the CPU bottleneck.
Please note: Firecrawl runs multiple workers (API + queue + 5 nuq workers + prefetch + index)