VM instance on GCP becomes unresponsive
Since I setup Novu with docker on a vm instance in gcp, without any real traffic, I have noticed that the vm becomes unresponsive after a couple of days, I recently increased storage to 250gb to check if it's a storage issue, and also increased the vm memory to 8gb and today the vm is unresponsive again, I am unable to ssh into it, and typically have to shutdown and start up again to regain access.
has anyone else experienced this? any directions as to what could possibly be going wrong?
11 Replies
@harrisyn, you just advanced to level 2!
I'm sorry to hear that you're experiencing issues with your VM becoming unresponsive. While I couldn't find an exact match to your issue, there have been reports of similar problems related to memory usage in Novu:
1. In one case, a Web Docker container experienced a "JavaScript heap out of memory" issue. The error was caused by insufficient RAM available for the Docker daemon during the container initialization. A deploy was made to help alleviate this issue in the prebuilt 0.11.0 version. There was also a suggestion to switch to NGINX for the web and widget Docker files for better memory management
https://github.com/novuhq/novu/issues/1950
https://discord.com/channels/895029566685462578/1036877046351138848
2. Another issue reported unoptimized containers greater than 1GB in size, which might cause problems depending on the available resources.
https://github.com/novuhq/novu/issues/1958
Recommendations to optimize the containers include using node16:alpine, multi-stage production builds, silent installs during module installation, and pruning dev dependencies.
Let me know if it helped 🙂
GitHub
🐛 Bug Report: Web Docker container gets existed with JavaScript hea...
📜 Description When using self-hosted version of Novu, many times docker container for Web gets terminated with error FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memor...
GitHub
🐛 Bug Report: Containers are not currently optimized and are >1GB ·...
📜 Description The issues with the current docker containers are: include dev dependencies do not use production pnpm builds include python of which is not used in the system they do not optimize th...
thanks for the response @emilpearce I will suspect (for now that it is due to low ram and increase it) however are there any recommendations for the host vm spec i.e. CPU, ram etc.
I am concerned that it is happening without any real traffic.
@dimagrossman @zacclifton Please have a look here
How did you deploy Novu to your vm? did you just run the docker compose on that server?
ran the compose and proxied behind nginx
I am setting up a new vm to rule out the vm being the problem, so a new install basically, will feedback
Ok, I setup on a new VM and ran into the same issue after 4 days.
@harrisyn, you just advanced to level 3!
Looks like the storage gets filled up, I had to scale the CPU to regain access as well
Important to state that this setup isn't in use, since I have unresolved CORS #Region is missing stated here, so no clients or notifications have either been setup or sent so far.
Any tips around what could be using up disk space will be appreciated
#My Ubuntu EC2 storage space is Getting Full
These are my findings.
Space was used up due to docker logs.
I had to truncate all docker logs to regain 90% of the storage
I assumed that setting env to production will limit log generation (Is there some other setting that handles this).
A better fix will be to update the docker-compose files with logging options or setup logrotate to rotate the logs automatically in the VM.
I can only go with logrotate, since I don't want to make local changes to the docker-compose file.
It would be great to have that as part of the repo, and use env variables to control the limits per service so logs for other services can be maintained.
e.g docker compose file
version: '3.7'
services:
app:
image: ...
logging:
options:
max-size: "10m"
max-file: "3"
If you want to put this in a PR we would be happy to merge it.
feat: add logging and restart to docker compose #3697
done