N
Novu•3w ago
notificationsdev

Email Processing is really slow during high load

We have a self hosted novu at our company. Under high load, we are seeing extremely slow processing -- almost 20 mins to run a worklow at times. We need emails to be instantaneous, so need help on improving this. We tried boosting the number of pods and give more CPU, but to no luck. How can we make the processing faster? Some errors we see: 
Unable to reach Bridge Endpoint at <URL>
Unable to reach Bridge Endpoint at <URL>
Under high load, the calls to bridge seems to time out: 
{"wait":0,"dns":8,"total":1012}}}},"name":"BridgeRequestError","bridgeError":{"type":"Object","message":"Unable to reach Bridge Endpoint at `https://novu-api.<company>-staging.com/v1/environments/<envId>/bridge`. Run npx novu@latest dev in Local mode, or ensure your Bridge app deployment is available.: ","stack":"\ncaused by: RequestError\n    at ClientRequest.<anonymous> (/usr/src/app/node_modules/.pnpm/got@11.8.6/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:633:26)\n    at ClientRequest.emit (node:events:530:35)\n    at origin.emit (/usr/src/app/node_modules/.pnpm/@szmarczak+http-timer@4.0.6/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at emitErrorEvent (node:_http_client:101:11)\n    at TLSSocket.socketErrorListener (node:_http_client:504:5)\n    at TLSSocket.emit (node:events:518:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)AggregateError [ETIMEDOUT]: \n    at internalConnectMultiple (node:net:1122:18)\n    at internalConnectMultiple (node:net:1190:5)\n    at Timeout.internalConnectMultipleTimeout (node:net:1716:5)\n    at listOnTimeout (node:internal/timers:583:11)\n    at process.processTimers 
{"wait":0,"dns":8,"total":1012}}}},"name":"BridgeRequestError","bridgeError":{"type":"Object","message":"Unable to reach Bridge Endpoint at `https://novu-api.<company>-staging.com/v1/environments/<envId>/bridge`. Run npx novu@latest dev in Local mode, or ensure your Bridge app deployment is available.: ","stack":"\ncaused by: RequestError\n    at ClientRequest.<anonymous> (/usr/src/app/node_modules/.pnpm/got@11.8.6/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:633:26)\n    at ClientRequest.emit (node:events:530:35)\n    at origin.emit (/usr/src/app/node_modules/.pnpm/@szmarczak+http-timer@4.0.6/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at emitErrorEvent (node:_http_client:101:11)\n    at TLSSocket.socketErrorListener (node:_http_client:504:5)\n    at TLSSocket.emit (node:events:518:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)AggregateError [ETIMEDOUT]: \n    at internalConnectMultiple (node:net:1122:18)\n    at internalConnectMultiple (node:net:1190:5)\n    at Timeout.internalConnectMultipleTimeout (node:net:1716:5)\n    at listOnTimeout (node:internal/timers:583:11)\n    at process.processTimers
And, it seems the calls to the bridge times out as well. Would appreciate guidance here.
1 Reply
Pawan Jain
Pawan Jain•3w ago
Pranish

Please use #community-self-host channel for self hosting issue

Did you find this page helpful?