CA
deep-jade
How to utilize memory in Apify actor which holds crawlee program with cheerio crawler?
Here I have attached a screenshot which is related to my last run. Within this run I have seen 700MB memory amount is in idle while the program execute. I have allocated 1024 memory to the actor. Also CPU usage looks very low. Furthermore, I have seen there are only maximum 15 concurrent requests have been running on the thread pool. Is there any way to utilize these factors?
4 Replies
useful-bronze•3y ago
Hi @Lahiru Gayantha, what actor do you use? Is it your own actor or an actor from the store?
deep-jadeOP•3y ago
It is my own actor.
useful-bronze•3y ago
Not sure what your crawler configuration looks like now, but you might be limiting the maxConcurrency there. You can also set the desiredConcurrency option on autoscaled pool options, which sets the initial concurrency of the crawler to this value (otherwise it starts at 1 and scales up).
https://crawlee.dev/docs/guides/scaling-crawlers#minconcurrency-and-maxconcurrency
https://crawlee.dev/api/core/interface/AutoscaledPoolOptions#desiredConcurrency
Scaling our crawlers | Crawlee
To infinity and beyond! ...within limits
deep-jadeOP•3y ago
@vojtechmaslan Thank you very much for the information.