How can I make sure that my self-hosted firecrawl can handle large amount of request?

If the request is plenty how could I handle the large amounts of requests?

6 Replies

you can simply increase the limits and configure it within the docker-compose.yaml example if you want to increase it for api worker:

deploy:
      resources:
        limits:
          cpus: '4.0'      # Increase if you have more CPU cores
          memory: 8G       # Increase if you have more RAM
        reservations:
          cpus: '2.0'      # Minimum guaranteed CPU
          memory: 4G       # Minimum guaranteed RAM

deploy:
      resources:
        limits:
          cpus: '4.0'      # Increase if you have more CPU cores
          memory: 8G       # Increase if you have more RAM
        reservations:
          cpus: '2.0'      # Minimum guaranteed CPU
          memory: 4G       # Minimum guaranteed RAM

Add this, and ensure your system (docker instance) has enough memory to spin up a bigger instance.

edsaurOP•2w ago

Ohh so its more on the docker-server that I need to set to handle concurrency right, sir @Gaurav Chadha? I need to increase the limits?

Gaurav Chadha•2w ago

yes, only if you want to scale, first check your current firecrawl container usage instead.

edsaurOP•2w ago

This is my container so its so far good right?

Gaurav Chadha•2w ago

Yea more than enough

edsaurOP•2w ago

But what if I want to limit the jobs how could we do it? Is the delay parameter enough?

Gaming

Programming

How can I make sure that my self-hosted firecrawl can handle large amount of request?

Did you find this page helpful?