Novu•3y ago

Self-hosted migration scripts in docker

Hi all!
We have self-hosted Novu in docker (docker-compose).
In the description of version 0.18 states that we have to execute the migration script.

npm run migration:primary-provider

npm run migration:primary-provider

npm run migration:primary-provider

npm run migration:primary-provider
What is the correct way to do it in docker-compose instance of Novu.

harrisyn•9/1/23, 10:43 AM

I have the exact same question

harrisyn•9/1/23, 10:44 AM

Did you manually change the image reference to 0.18 or you are only preparing for the update

Dmitry AbramovOP•9/4/23, 4:12 PM

aegeavaz•9/5/23, 10:57 AM

I'm running with the same issue. I tried to upgrade the api service in docker-compose to 0.18, but the migration doesn't exist in that version (even in the source code that migration doesn't exist in the package.json file in the api folder.
So I checked in the 0.19 version, and there is it. So I tried to upgrade the docker image to 0.19 and run the migration inside the container but it failed because reference another file in the src folder.

So it looks like the migration only works from the source code, and not from the containers.
Lets see what Novu staff says about this.

Thanks

Dima Grossman•9/5/23, 11:17 AM

Hi @aegeavaz and sorry for the delayed response here, I would suggest running the migrations from source and not from the containers. (You can clone it locally or in a machine that has access to the mongo instance, and run the migrations)

DDima Grossman Hi @aegeavaz and sorry for the delayed response here, I would suggest running th...

aegeavaz•9/5/23, 12:08 PM

Thanks. But what about the versions? because that migration is not in the v.0.18.0 tag version but it is described in the release documentation for v.0.18.0. Should we upgrade to version 0.19.0?

Dima Grossman•9/5/23, 12:49 PM

You can just clone the next branch, all the migrations file are there historically

DDima Grossman You can just clone the next branch, all the migrations file are there historical...

Dmitry AbramovOP•9/7/23, 4:48 AM

Well, it works. Thank you!

harrisyn•9/12/23, 10:09 AM

@dimagrossman

@novu/api@0.19.0 migration
cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/expire-at/expire-at.migration.ts

node:events:491
throw er; // Unhandled 'error' event
^

Error: spawn ts-node ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:289:12)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn ts-node',
path: 'ts-node',
spawnargs: [
'--transpileOnly',
'./migrations/expire-at/expire-at.migration.ts'
]
}

Node.js v18.7.0
npm ERR! Lifecycle script migrationmigration failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/api@0.19.0
npm ERR! at location: /home/technology/novu/apps/api

harrisyn•9/12/23, 10:10 AM

this is what I get

harrisyn•9/12/23, 10:10 AM

I am running this against the next branch

harrisyn•9/13/23, 11:46 AM

any insights to resolve this?

Dima Grossman•9/14/23, 7:22 AM

@harrisyn based on your logs you try to run the migrationmigration npm script, i'm not sure what version you are upgrading to, but in 0.18.0 the script is:

npm run migration:primary-provider

npm run migration:primary-provider

DDima Grossman @harrisyn based on your logs you try to run the `migration` npm script, i'm not ...

harrisyn•9/14/23, 8:01 AM

Hi @dimagrossman
I am running this off the next branch, so you can see the workspace is @novu/api@0.19.0
might have been assumed, however I think node_modules have to be installed first, this environment had only docker.
What I have tried
1) Ran npm i from the apps/api directory
2) ran npm i from the base novu folder

I still end up now with

npm run migration:primary-provider

@novu/api@0.19.0 migration:primary-provider
cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/integration-scheme-update/add-primary-priority-migration.ts

Error: Cannot find module '/home/technology/novu/node_modules/@novu/shared/dist/cjs/index.js'. Please verify that the package.json has a valid "main" entry
at tryPackage (node:internal/modules/cjs/loader:364:19)
at Function.Module._findPath (node:internal/modules/cjs/loader:577:18)
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:942:27)
at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue [as _resolveFilename] (/home/technology/novu/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30)
at Function.Module._load (node:internal/modules/cjs/loader:804:27)
at Module.require (node:internal/modules/cjs/loader:1022:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.<anonymous> (/home/technology/novu/apps/api/src/config/index.ts:2:1)
at Module._compile (node:internal/modules/cjs/loader:1120:14)
at Module.m._compile (/home/technology/novu/node_modules/ts-node/src/index.ts:1618:23) {
code: 'MODULE_NOT_FOUND',
path: '/home/technology/novu/node_modules/@novu/shared/package.json',
requestPath: '@Novu/shared'
}
npm ERR! Lifecycle script migration:primary-providermigration:primary-provider failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/api@0.19.0
npm ERR! at location: /home/technology/novu/apps/api

Dima Grossman•9/14/23, 8:10 AM

@harrisyn could you please run: pnpm install && npm run build

From the root repository folder

Dima Grossman•9/14/23, 8:10 AM

pnpm part is important

harrisyn•9/21/23, 11:14 AM

Hi @dimagrossman I am still not successful at this.

harrisyn•9/21/23, 11:55 AM

What I have tried

1) updated the codebase to the latest in next branch.
2) installed pnpm and node (it didn't work with the latest versions, so installed the recommended versions)
3) ran pnpm install && npm run build

Successfully ran target prebuild for 60 projects (49s)

nx run @Novu/web:build

@novu/web@0.19.0 build /home/technology/novu/apps/web
cross-env NODE_OPTIONS=--max_old_space_size=4096 react-app-rewired --max_old_space_size=4096 build

Creating an optimized production build...
The build failed because the process exited too early. This probably means the system ran out of memory or someone called kill -9kill -9 on the process.
ELIFECYCLE Command failed with exit code 1.

————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

NX Ran target build for 65 projects (38m)

57/59 succeeded [0 read from cache]

2/59 targets failed, including the following:

nx run @Novu/outlook365:build
nx run @Novu/web:build

this process takes waaaaay too long, should I need to increase the VM specifications to run a migration??

Dima Grossman•9/21/23, 1:34 PM

I think the issue is with the web projects, which in reality arent required to be built for the migrations

Dima Grossman•9/21/23, 1:35 PM

could you try doing: npm run build:api insteald of the global build

Dima Grossman•9/21/23, 1:36 PM

And by the way, i'm very sorry for the very complicated flow on this. I would really love to simplify this. Just hadn't a change to get to it yet

harrisyn•9/21/23, 4:35 PM

oot@Novu:/home/technology/novu# npm run build:api

build:api
nx build @Novu/api

1/47 dependent project tasks failed (see below)

46/47 dependent project tasks succeeded [46 read from cache]

nx run @Novu/outlook365:build

@novu/outlook365@0.19.0 build /home/technology/novu/providers/outlook365
run-p build:*

@novu/outlook365@0.19.0 build:main /home/technology/novu/providers/outlook365
tsc -p tsconfig.json

@novu/outlook365@0.19.0 build:module /home/technology/novu/providers/outlook365
tsc -p tsconfig.module.json

src/lib/outlook365.provider.ts(24,7): error TS2769: No overload matches this call.
The last overload gave the following error.
Argument of type '{ host: string; port: number; requireTLS: boolean; connectionTImeout: number; auth: { user: string; pass: string; }; tls: { ciphers: string; }; }' is not assignable to parameter of type 'TransportOptions | Transport<any>'.
Object literal may only specify known properties, and 'host' does not exist in type 'TransportOptions | Transport<any>'.
src/lib/outlook365.provider.ts(24,7): error TS2769: No overload matches this call.

NX Ran target build for project @Novu/api and 48 task(s) they depend on (11s)

1/47 failed

46/47 succeeded [46 read from cache]

harrisyn•9/26/23, 9:58 AM

@Dima Grossman any alternatives for me?
currently stuck here

harrisyn•10/4/23, 12:59 AM

https://cloud.nx.app/runs/rwevxDlyAn

Distributed Caching and Task Execution for Nx and Lerna

Leverage the power of distribution to bring your Nx and Lerna workspaces to the next level. All nicely integrated into a single unified experience.

harrisyn•10/4/23, 12:59 AM

@Dima Grossman

todd•10/4/23, 1:21 AM

@Dima Grossman I am following this thread and wondering how I am going to do this inside a build pipeline with containers in locked down networks.

How do you do it for your production environments?

Cheers.

Ttodd @Dima Grossman I am following this thread and wondering how I am going to do thi...

Dima Grossman•10/4/23, 7:10 AM

Hi @todd, in this case you will need to have a virtual machine running in the vpc you are referring to.

Dima Grossman•10/4/23, 7:15 AM

I really don't know what to say. We are running this in CI multiple times a day and never have encountered this error. Also tried to replicate this locally with a clean git clone. I have a hunch with something that might be with nodemailer types there, but it still doesn't explain why this happening for you and in none of my attempts to replicate.

Doing a small test, and then will ask you you try fetching again from next

Dima Grossman•10/4/23, 7:18 AM

Alright, managed to replicate the error after installing "@types/nodemailer" in the outlook 365 package.

Might be you have some globally hoisted types package present in your environment. issuing a fix right now. Hope it will solve it

Dima Grossman•10/4/23, 8:58 AM

@harrisyn pushed a change to next, could you try pulling again and running the build to see if it was resolved?

harrisyn•10/4/23, 9:12 AM

Will give it a shot, however I tried to debug last night and found
1) outlook365 -> connectionTImeout (typo) should be connectionTimeout - this built successfully with this change
2) errors on line 45 and 48 in packages/application-generic/src/utils/filter.ts, Property 'includes' does not exist on type 'unknown'. -> added a check if it's not null or undefined and it was fine

next error is in passport strategy. I gave up at this point

DDima Grossman @harrisyn pushed a change to next, could you try pulling again and running the b...

harrisyn•10/4/23, 9:55 AM

@novu/api@0.20.0-alpha.0 build /home/technology/novu/apps/api
nest build

ERROR in /home/technology/novu/apps/api/src/app/auth/services/passport/github.strategy.ts
19:19-23
[tsl] ERROR in /home/technology/novu/apps/api/src/app/auth/services/passport/github.strategy.ts(19,20)
TS2345: Argument of type 'null' is not assignable to parameter of type 'Error'.
ts-loader-default_43d7ed5e51b1fe49
webpack 5.88.2 compiled with 1 error in 49811 ms
ELIFECYCLE Command failed with exit code 1.

harrisyn•10/4/23, 9:55 AM

@Dima Grossman same spot, synced with next

Hharrisyn Will give it a shot, however I tried to debug last night and found 1) outlook365...

Dima Grossman•10/4/23, 10:46 AM

1) totally correct, also pushed this to next.
2) Do you have some node modules located in your technology folder or home folders?

It feels like your compilation happens in some other ts environment

harrisyn•10/4/23, 11:57 AM

nope, that environment only had docker installed, I ran the npm run clean command a few times just to be sure and even before each attempt

Dima Grossman•10/4/23, 2:23 PM

Argh

@BiswaViraj could you maybe assit here to see if you are able to catch something? Seems like some how the ts compiler for @harrisyn get's access to @types packages that are not available for us during the build. Like in the previous example with nodemailer, or right now.

Just to verify you have only done

git clone https://github.com/novuhq/novu.git
cd novu
pnpm install
pnpm build

git clone https://github.com/novuhq/novu.git
cd novu
pnpm install
pnpm build

?

pnpm i is important and not to use npm directly for each package to properly install dependencies.

BiswaViraj•10/4/23, 2:38 PM

Looking into this

harrisyn•10/4/23, 3:48 PM

https://cloud.nx.app/runs/pkadanCggc

Distributed Caching and Task Execution for Nx and Lerna

Leverage the power of distribution to bring your Nx and Lerna workspaces to the next level. All nicely integrated into a single unified experience.

BiswaViraj•10/4/23, 3:56 PM

Have you pulled the latest changes from our next branch?

As I can see, it is trying to build docsdocs which we have removed in the latest version.

Please try to pull the latest changes from our nextnext branch then follow these steps:

remove the
```
node_modules
```
```
node_modules
```
folder
run npx nx resetnpx nx reset
run npm run setup:projectnpm run setup:project

DDima Grossman Hi @todd, in this case you will need to have a virtual machine running in the vp...

todd•10/4/23, 5:44 PM

Sorry, I see this is slightly off topic on this thread—self hosted looks to be probably locally hosted from source. My question is how/when is the migration triggered when running from pre-build images (yes, I have a cluster in a VPC).

I see the migration scripts in the api project. I see that scripts that run them manually (for a local migration).

What I am fundamentally trying to understand is do I need to worry about migrations for my production instance? It looks like I don't but I just can't see in code how the migration scripts are bootstrapped and when. By the api, by a worker? At bootstrap or some interval or other trigger?

Happy to be pointed at the code line and I'll work out the rest. Just struggling!

Keep up the great work

Novu_Bot•10/4/23, 5:44 PM

@todd, you just advanced to level 3!

Ttodd Sorry, I see this is slightly off topic on this thread—self hosted looks to be p...

BiswaViraj•10/4/23, 6:04 PM

Currently, we run our migrations manually using the script. As @Dima Grossman mentioned, you'll need to create a VM in your VPC and then just run those scripts

BBiswaViraj Currently, we run our migrations manually using the script. As @Dima Grossman me...

todd•10/4/23, 6:14 PM

Thanks. I wonder a couple of things. Again off topic in this thread and apologise.

There are three migration commands in that project. So, I wonder what you actually do as well—as in the entire process.

I suppose without this it hard to see that the project can really be run successfully by others in production. It seems to be shame to have a manual steps in deployment pipeline and I will be looking to ensure this is all scripted.

Thanks in advance. (I suspect this all looks obvious to you who do it every day but it is very opaque to me—and migrations are somewhat fundamental)

Ttodd Thanks. I wonder a couple of things. Again off topic in this thread and apologis...

BiswaViraj•10/4/23, 6:47 PM

I agree with you about having automated migrations, maybe we can include it in the docker

.
For choosing which migration command to run, I will recommend you to follow the version release guides on github as it will have the relevant details about which command to run.

BBiswaViraj I agree with you about having automated migrations, maybe we can include it in t...

todd•10/4/23, 6:58 PM

Looking at that documentation, I would like to make a distinction between Upgrades and Migrations. Upgrades definitely "manual" because you need are introducing new features/flags etc. Migrations to me are data migrations WITHIN the existing infrastructure. These are ideally transparent unless the migration infrastructure itself changes which is then an upgrade!

This distinction is all very apparent when the pipeline is fully "automated" (eg say terraform on a build pipeline)

In my own product, I have a migration as a lambda (AWS function with single entry point) that is triggered by the pipeline to run on every deployment (in every environment). So logically the same as first mentioned by Dima and I can do this here.

I just needed to work out the what and was hoping for explicit instructions!

Like I've done with the S3 setup, I will come back when I work out the finer details. I have already had to do this also with MongoDB Atlas.

Thanks for the help

Ttodd Looking at that documentation, I would like to make a distinction between Upgrad...

harrisyn•10/4/23, 10:04 PM

@todd definitely not off topic, as it was my initial assumption i.e. that migrations can be automated.
It should have been relatively easy if it could be run against a running container and be done - it's how I have done it in my other applications as part of the pipeline.

BBiswaViraj Have you pulled the latest changes from our next branch? As I can see, it is tr...

harrisyn•10/4/23, 10:14 PM

I can confirm that I am running off the latest, synced earlier today as well, so I saw the fixes I had highlighted previously have been effected into the next branch.

npx nx reset

root@Novu:/home/technology/novu# npx nx reset
node:internal/modules/cjs/loader:1031
throw err;
^

Error: Cannot find module '@nrwl/workspace/presets/npm.json'
Require stack:

/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/nx-json.js
/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/configuration.js
/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/utils/package-manager.js
/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/init-local.js
/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/nx.js
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1028:15)
at Function.resolve (node:internal/modules/cjs/helpers:125:19)
at readNxJson (/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/nx-json.js:13:48)
at Object.<anonymous> (/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/daemon/client/client.js:319:72)
at Module._compile (node:internal/modules/cjs/loader:1198:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
at Module.load (node:internal/modules/cjs/loader:1076:32)
at Function.Module._load (node:internal/modules/cjs/loader:911:12)
at Module.require (node:internal/modules/cjs/loader:1100:19)
at require (node:internal/modules/cjs/helpers:119:18) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/nx-json.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/config/configuration.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/src/utils/package-manager.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/init-local.js',
'/root/.npm/_npx/973fb08ee51bb939/node_modules/nx/bin/nx.js'
]
}

Hharrisyn I can confirm that I am running off the latest, synced earlier today as well, so...

Novu_Bot•10/4/23, 10:14 PM

Your roles on this server do not allow you to use the AI commands.

Reach out to the server owner to know how to get an ai-authorized role or Unlock unlimited uses of this AI-based feature with a personal plan.

BBiswaViraj Have you pulled the latest changes from our next branch? As I can see, it is tr...

harrisyn•10/4/23, 11:10 PM

got some progress with this.

However

novu# cd apps/api
root@Novu:/home/technology/novu/apps/api# npm run migration:primary-provider

@novu/api@0.20.0-alpha.0 migration:primary-provider
cross-env NODE_ENV=local MIGRATION=true ts-node --transpileOnly ./migrations/integration-scheme-update/add-primary-priority-migration.ts

New Relic for Node.js was unable to bootstrap itself due to an error:
Error: New Relic requires that you name this application!
Set app_name in your newrelic.js or newrelic.cjs file or set environment variable
NEW_RELIC_APP_NAME. Not starting!
at createAgent (/home/technology/novu/node_modules/.pnpm/newrelic@9.15.0/node_modules/newrelic/index.js:140:11)
at initialize (/home/technology/novu/node_modules/.pnpm/newrelic@9.15.0/node_modules/newrelic/index.js:81:15)
at Object.<anonymous> (/home/technology/novu/node_modules/.pnpm/newrelic@9.15.0/node_modules/newrelic/index.js:36:3)
at Module._compile (node:internal/modules/cjs/loader:1198:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
at Object.require.extensions.<computed> [as .js] (/home/technology/novu/node_modules/.pnpm/ts-node@10.9.1_wh2cnrlliuuxb2etxm2m3ttgna/node_modules/ts-node/src/index.ts:1608:43)
at Module.load (node:internal/modules/cjs/loader:1076:32)
at Function.Module._load (node:internal/modules/cjs/loader:911:12)
at Module.require (node:internal/modules/cjs/loader:1100:19)
at require (node:internal/modules/cjs/helpers:119:18)
Environment: local
Platform: Docker
Tenant: OS
Selected Log Transport pino-pretty
Migration for primary and priority fields in the integration entity

npm ERR! Lifecycle script migration:primary-providermigration:primary-provider failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: @novu/api@0.20.0-alpha.0
npm ERR! at location: /home/technology/novu/apps/api

harrisyn•10/4/23, 11:17 PM

I suspect the NEW_RELIC error can be ignored, however the migration didn't run successfully, I have added the .env in the same folder i.e. apps/api to ensure it can connect to my mongodb database

Self-hosted migration scripts in docker

Similar Threads

Similar Threads

Similar Threads