relationship between memoryMbytes and availableMemoryRatio?
Can someone tell me what the relationship between those two configuration options is? is one jut in MB and the other in a ratio of the total?
I thought availableMemoryRatio limited the total memory used, but "When the memory usage is more than the provided ratio, the memory is considered overloaded." so it's more like an indicator? I'm currently working on a heavily overloaded server, and would rather that crawlee/playwright had a strong upper limit and was slower.
10 Replies
graceful-blueOP•2y ago
and with overloaded I mean it's using 600% of the available memory 🙊
eager-peach•2y ago
availableMemoryRatio is not a memory limit but autoscaling config telling it when you scale up or down. What you probably want to set set the max memory for the whole process.graceful-blueOP•2y ago
yes, is there a way to do that?
eager-peach•2y ago
Configuration | Crawlee
Configuring Crawlee parameters
eager-peach•2y ago
You can set it as env var or in the code via Configuration class
graceful-blueOP•2y ago
I have not set that, but the docs mention this:
By default, the max amount of memory to be used is set to one quarter of total system memory, i.e. on a system with 8192 MB of memory, the autoscaling feature will only use up to 2048 MB of memory.yet it's going well beyond 25% of the available memory, so it doesn't look like it limits memory usage
@Kilian just advanced to level 3! Thanks for your contributions! 🎉
eager-peach•2y ago
Its does not really put a hard limit, it will just not scale up in that case. If you are doing a lot of work inside a handler or there is sudden spike, it can go up.
graceful-blueOP•2y ago
so there's no way to hard-cap it?
eager-peach•2y ago
You can do that on Node.js level, see https://support.circleci.com/hc/en-us/articles/360009208393-How-Can-I-Increase-the-Max-Memory-for-Node-
CircleCI Support Center
How Can I Increase the Max Memory for Node?
Default Memory Size
Node has a default max memory usage of less than 2GB on some 64bit systems (depending on the Node runtime version). This can cause unexpected memory issues when running on Circl...