SpooKyHammer - Hello, what's the recommend spec...
Hello, what's the recommend spec to self host firecrawl? I am currently trying it out 2 vCPU and 4GB ram but its crashes the server after running successful crawl on 20-30 websites. and does it internally cache things if so how can I disable it?
3 Replies
Hi @SpooKyHammer Your 2 vCPU and 4GB RAM setup is likely hitting, what is the error on logs?
Recommended Steup:
- 4+ vCPU (Firecrawl runs multiple worker processes: API server, queue-worker, 5x nuq-workers, nuq-prefetch-worker, index-worker)
- 8+ GB RAM (to handle browser automation via Playwright, Redis, PostgreSQL, and concurrent scraping jobs)
From logs I can see OOM killed, but what's interesting is it was able to work for first 20-30 websites and then fail rather than fail at the beginning itself
Seems a memory leak issue to me, Try to set NUQ_WORKER_COUNT=2 in your .env to reduce parallel workers from 5 to 2. This significantly reduces memory footprint