N
Novuβ€’4w ago
MP

Slowness

Hi Team, I'm experiencing slowness in Novu, with DB utilization nearing 100%. We are using 8GB of CPU for the database and 4GB for the application server, with 2 workers. The application server load is minimal, only 10-30% at most. Our daily notification count is 14 thousand, distributed throughout the day. What steps should I take to resolve this issue? What will be the impact if I increase QUEUE_CHUNK_SIZE? Thanks
No description
10 Replies
Ask Inkeep
Ask Inkeepβ€’4w ago
Hi @MP, I'm an AI assistant for Novu. While a team member reviews this question, I'll see if I can find relevant sources for your question. ----------------------------- While I generate an answer, here are some sources that might be helpful: * Get the unseen in-app notifications count for subscribers feed * Introduction * Introduction to Novu * Run Novu in local machine ----------------------------- Answer It seems that your issue with high database utilization might be related to insufficient resources allocated to your MongoDB instance, particularly if you are experiencing high CPU usage. Based on the information provided, here are a few steps you can consider to address the slowness: 1. Index Optimization: Ensure that your MongoDB instance is properly indexed. Missing indexes can lead to high CPU usage as the database performs full scans over the data. You might want to review your MongoDB logs to identify slow queries and ensure that appropriate indexes are in place.
2. Increase Resources: Given that your database utilization is nearing 100%, increasing the resources allocated to your MongoDB instance might help. This could involve upgrading the instance to a higher specification with more CPU and RAM. 3. Review Worker Configuration: Since you mentioned using 2 workers, consider whether the distribution of tasks to these workers is optimal. Balancing the load more effectively across workers might help reduce the strain on your database. 4. Adjust QUEUE_CHUNK_SIZE: Increasing the QUEUE_CHUNK_SIZE can have implications depending on your current setup. This setting determines the number of jobs fetched from the database in one go. Increasing it might reduce the number of queries made to the database, potentially lowering CPU usage if your bottleneck is query frequency. However, it could also increase memory usage on the workers if they fetch more data than they can efficiently process. 5. Monitoring and Logs: Implement more detailed monitoring on your MongoDB instance to get real-time insights into performance bottlenecks. Tools like MongoDB Atlas, if you are using it, provide valuable metrics on performance and can help in identifying issues with slow queries. 6. Database Scaling: If the load on your database continue s to increase, consider scaling your MongoDB deployment. This could involve scaling vertically (more powerful hardware) or horizontally (adding more nodes). As for the impact of increasing the QUEUE_CHUNK_SIZE, it could potentially reduce the frequency of database queries, which might help with CPU utilization. However, this should be tested carefully as it could also lead to increased memory consumption on the workers, which might offset some of the benefits if not managed properly. These suggestions are based on general best practices for managing database performance issues and scaling workers in a distributed system. For a more tailored solution, it might be helpful to consult detailed logs and performance metrics from your environment. Got it. Tagging @Support for additional help.
MP
MPβ€’4w ago
Additionally, MongoDB queries in jobs are taking a long time. @Support
No description
Novu_Bot
Novu_Botβ€’4w ago
@MP, you just advanced to level 2!
Tomer Barnea
Tomer Barneaβ€’4w ago
Hey @MP, let's see if @Pawan Jain @Dima Grossman, @sokratis can help you here. Can you share a bit about the way you use Novu?
Dima Grossman
Dima Grossmanβ€’4w ago
Hi @MP my first suspicion would be missing indexes for the queries. What indexes do you currently have for the jobs collection?
MP
MPβ€’4w ago
Hi @Tomer Barnea, We are using Novu to notify our users Class is booked or Cancelled and also some reminders emails about class. this reminders workflow will run from every 30 minutes to notify users who has future bookings @Dima Grossman please find the indexes what we have for jobs collection
No description
Dima Grossman
Dima Grossmanβ€’4w ago
Yep it defiently seems like you are missing indexes in your database take a look at job.schema.ts file in @novu/dal package, it has a list of suggested indexes to incorporate depending on your usecases
MP
MPβ€’4w ago
Thanks @Dima Grossman , Is there any other places where I can add indexes to improve performance? Shall I increase the QUEUE_CHUNK_SIZE from default?
Dima Grossman
Dima Grossmanβ€’4w ago
On all Schemas πŸ™‚ @MP you can try increasing it, to see if this improves performance for you are you using any observability platform like NewRelic? This can be helpful to diagnose performance issues
MP
MPβ€’3w ago
Thanks for the support @Dima Grossman πŸ˜‡
Want results from more Discord servers?
Add your server
More Posts
depoly novu using docker compose and nginxI'm trying to deploy novu (production) using docker compose and nginx, I got: > https://notify.examEcho Workflow VariantsHi all I would like to try out Echo and was looking for a way how to specify variants in the docs, Does novu support sending messages to slack in markdownI want to send the message in table form in a slack channel via novu. How can this be achieved?Cannot open 'Layouts' tabWhen I click on Layouts on the novu dashboard, my page just blinks and go back to Workflows. I triedDoes the notification widget for React support WS service on a path?I'm deploying the self-hosted version to my infrastructure and have put API and WS services on a patNovu Headless - notifications not showing for custom subscriber IDsI have an app that accepts user's first name, last name, and first name. Then, it generates an ID siNot able to see the WEB UI on 4200I have been trying to setup the novu on my local machine after too many tries. i was able to run apiHow can I integrate in-app notifications in React Native?The Novu documentation contains integration guides for 'React' and other web apps. I need to integraGetting error "POST https://mydomain.com/api/v1/widgets/session/initialize 404 (Not Found)my .env example # Root URL REACT_APP_WS_URL=$HOST_NAME/ws API_ROOT_URL=$HOST_NAME/api DISABLE_USER_Firebase is forcing users to migrate from FCM to HTTP v1 (Does Novu supports HTTP v1)??Hi everyone I've got an email from firebase (check the attached email on the bottom) And I was wond