CA
Crawlee & Apify•3w ago
deep-jade

Memory is critically overloaded

I have an AWS EC2 instance with 64GB memory. My crawler is running in a docker container. The CRAWLEE_MEMORY_MBYTES is set to 61440 My docker config 
docker run --rm --init -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e SPAWN=${SPAWN} \
    -e MONGO_CACHE=${MONGO_CACHE} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args
docker run --rm --init -t $docker_args \
    -v /mnt/storage:/app/storage \
    --user appuser \
    --security-opt seccomp=/var/lib/jenkins/helpers/docker/seccomp_profile.json \
    -e MONGO_HOST=${MONGO_HOST} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e SPAWN=${SPAWN} \
    -e MONGO_CACHE=${MONGO_CACHE} \
    -e CRAWLEE_MEMORY_MBYTES=${CRAWLEE_MEMORY_MBYTES} \
    ${IMAGE_NAME} $prog_args
After crawling around 15-20 minutes I receive the warning Memory is critically overloaded. But when doing the command free -h I notice I still have a lot of free memory as you can see in the screenshots. The playwright crawler is configured like this 
 crawler = PlaywrightCrawler(
        concurrency_settings=ConcurrencySettings(
            min_concurrency=25,
            max_concurrency=125,
        ),
        request_manager=request_queue,
        request_handler=router,
        retry_on_blocked=True,
        headless=True,
    )
 crawler = PlaywrightCrawler(
        concurrency_settings=ConcurrencySettings(
            min_concurrency=25,
            max_concurrency=125,
        ),
        request_manager=request_queue,
        request_handler=router,
        retry_on_blocked=True,
        headless=True,
    )
I'm a bit clueless why I have this issue. In the past I had issues with zombie processes but this was solved by adding the --init to my docker run command. Do you have any idea on how I can fix this or further debug this?
No description
No description
3 Replies
Hall
Hall•3w ago
Someone will reply to you shortly. In the meantime, this might help:
passive-yellow
passive-yellow•3w ago
Hi @ROYOSTI I recommend you post this situation in the repository, as it could be either another one of the customization features for AWS or a bug in the resource counter.
conventional-tan
conventional-tan•2w ago
Could you give us some more info? Specifically what playwright version you're using 👀 And could you try with older playwright, maybe its an issue with the newer versions

Did you find this page helpful?