CA
Crawlee & Apify•3y ago
rival-black

How to increase max memory?

I am running a script that needs concurrency. I have 64 GB of RAM available and I want to use it to the max. I am running my script on a server so there is not much else running. The problem is, at around 15GB I always get memory overloaded error. I have tried: 
config.set('memoryMbytes', 50_000)
config.set('availableMemoryRatio', 0.95)
config.set('memoryMbytes', 50_000)
config.set('availableMemoryRatio', 0.95)
Nothing seems to change this behavior. Anything else I can try?
1 Reply
Pepa J
Pepa J•3y ago
Hello Jeno, based on https://sdk.apify.com/api/apify/interface/ConfigurationOptions#memoryMbytes it seems to me, that in your case, there is still the default memory limit being used. Have you tried to run the script with CRAWLEE_MEMORY_MBYTES environmental variable being set?

Did you find this page helpful?