Novu•3y ago

Issue while having both FCM and APNS active

Hi all.

I'm facing a strange issue while having both FCM and APNS integrations set and active.
I'm setting the credentials for a subscriber with an APNS token like this:

{
    "providerId": "apns",
    "credentials": {
        "deviceTokens": ["xxxxxx"]
    }
}

{
    "providerId": "apns",
    "credentials": {
        "deviceTokens": ["xxxxxx"]
    }
}

{
    "providerId": "apns",
    "credentials": {
        "deviceTokens": ["xxxxxx"]
    }
}

{
    "providerId": "apns",
    "credentials": {
        "deviceTokens": ["xxxxxx"]
    }
}

So the subscriber has a token related to apns. But when I'm trying to send a push notification to that user using an existing template, I'm getting FCM related errors.

{
  "stack": "Error: Sending message failed due to \"The registration token is not a valid FCM registration token\"\n    at FcmPushProvider.<anonymous> (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:112:35)\n    at step (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:44:23)\n    at Object.next (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:25:53)\n    at fulfilled (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:16:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
  "message": "Sending message failed due to \"The registration token is not a valid FCM registration token\""
}

{
  "stack": "Error: Sending message failed due to \"The registration token is not a valid FCM registration token\"\n    at FcmPushProvider.<anonymous> (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:112:35)\n    at step (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:44:23)\n    at Object.next (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:25:53)\n    at fulfilled (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:16:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
  "message": "Sending message failed due to \"The registration token is not a valid FCM registration token\""
}

{
  "stack": "Error: Sending message failed due to \"The registration token is not a valid FCM registration token\"\n    at FcmPushProvider.<anonymous> (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:112:35)\n    at step (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:44:23)\n    at Object.next (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:25:53)\n    at fulfilled (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:16:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
  "message": "Sending message failed due to \"The registration token is not a valid FCM registration token\""
}

{
  "stack": "Error: Sending message failed due to \"The registration token is not a valid FCM registration token\"\n    at FcmPushProvider.<anonymous> (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:112:35)\n    at step (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:44:23)\n    at Object.next (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:25:53)\n    at fulfilled (/usr/src/app/providers/fcm/build/main/lib/fcm.provider.js:16:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
  "message": "Sending message failed due to \"The registration token is not a valid FCM registration token\""
}

Do I miss a step of the process?

Thanks!

Pawan Jain•3/22/23, 12:09 PM

As you have both providers active,
Did you add fcm deviceToken also for this particular subscriber?

serafeimoOP•3/22/23, 1:20 PM

No, I only have apns token set.

Pawan Jain•3/22/23, 1:21 PM

I guess that's the reason of this. It requires valid FCM token as well if FCM is active

serafeimoOP•3/22/23, 1:39 PM

I don't think that it makes sense though. There are users that have android devices and others with ios devices and we need to send notifications to all of them. I guess that this is a pretty common case in general.

Pawan Jain•3/22/23, 2:08 PM

Ohh! I got your usecase.

Pawan Jain•3/22/23, 2:09 PM

So one solution I can think of is, there should be option to activate/deactivate any of provider dynamically during trigger

serafeimoOP•3/22/23, 2:22 PM

I can't see an option in trigger endpoint. I can see only an API to update an integration (i.e. set it on/off) but in any case this will be an overkill to do for every notification. In addition, I'm sure that it will create problems when many notifications will being sent in parallel.

serafeimoOP•3/23/23, 11:19 AM

I've tested it further and it seems that it tries to send notifications using every active integration, regardless of the subscriber's credentials. So, for example:

FCM -> active
APNS -> active
Subscriber has only FCM token set in credentials.
Send push notification
2 actions have been triggered:
- sent with FCM successfully
- failed to send with APNS (due to lack of token)

-- From my point of view, the expected scenario would be to send one notification using FCM since this is what the subscriber has credentials for.

Of course the above example is the same for the other way around (only apns token).

I believe that the provider should be used according to the subscriber's credentials so that the system can send notifications to users that have only android or ios devices without the need of constantly activating/deactivating the providers. I think it's the normal and most common case and not an exceptional scenario.

LinearAPP•3/28/23, 5:11 PM

Issue NV-1924 created.

NV-1924 - notification failing due to absence of token in case of multiple active push providers

Status

Triage

Engineering

•

3/28/23, 5:11 PM

serafeimoOP•3/29/23, 7:13 AM

Good morning!
Another thing I noticed is the following:

Activate FCM
Activate APNS
Create a subscriber with both FCM and APNS tokens
When sending push notifications, Novu tries to send both notifications with FCM. So, for the second token it will get an error because it's not a valid FCM token but an APNS one. It looks like it gets the first active integration and use it for everything.

Pawan Jain•3/29/23, 7:22 AM

and both tokens are valid tokens

serafeimoOP•3/29/23, 7:39 AM

Yes, FCM valid token and APNS valid token

serafeimoOP•3/29/23, 7:41 AM

So, it seems that when I have 2 tokens set in the subscriber, Novu uses FCM for both, regardless of the APNS's provider state (active/inactive)

Pawan Jain•4/5/23, 6:45 AM

It's not on GitHub
It is in triage in linear.

frnandu•9/29/23, 9:24 AM

Any updates on this?

frnandu•9/29/23, 9:24 AM

Is this the same as https://github.com/novuhq/novu/issues/3797 ?

GitHub

🐛 Bug Report: Wrong provider is selected when sending a push notifi...

Description In the case of the push notifications, given that we have 2 active providers in the same organization: fcm (for android and web push notifications) and apns (for ios notifications), w...

Pawan Jain•9/29/23, 10:03 AM

@frnandu
Currently, override is possible in sms and email providers only

Pawan Jain•9/29/23, 10:04 AM

We are working on addding this option for chat and push channel providers

frnandu•9/29/23, 10:05 AM

Any idea on expected time?

frnandu•9/29/23, 10:06 AM

Without this, I will be forced to make separate workflows for fcm/apns and handle that on backend, which is a pain...

Pawan Jain•10/2/23, 12:47 PM

@frnandu
I have shared this with team
I will revert back with possible ETA

frnandu•10/3/23, 1:03 PM

Even with separate workflows, it is not possible to use fcm provider for one workflow and apns for another. This basically makes it impossible to use simultaneously PUSH for fcm and apns even if you'd filter on backend for different workflows depending on user having credentials (deviceTokens) for android or ios. Please make this a top priority on your team, otherwise Novu becomes unable of handling PUSH's for mixed users of android/ios.

frnandu•10/23/23, 11:22 AM

This https://github.com/novuhq/novu/issues/3797 was marked as fixed/closed but I've just tested 0.20.0 docker images and the problem is still happening.

GitHub

[NV-2935] 🐛 Bug Report: Wrong provider is selected when sending a p...

Description In the case of the push notifications, given that we have 2 active providers in the same organization: fcm (for android and web push notifications) and apns (for ios notifications), w...

Novu_Bot•10/23/23, 11:22 AM

@frnandu, you just advanced to level 2!

Ffrnandu This https://github.com/novuhq/novu/issues/3797 was marked as fixed/closed but I...

--•10/23/23, 12:13 PM

Can you try setting in the environment variables of the API, Worker and Web apps the following environment variable:

IS_MULTI_PROVIDER_CONFIGURATION_ENABLED='true'

IS_MULTI_PROVIDER_CONFIGURATION_ENABLED='true'

This will enable the multi provider feature that includes the fix of the bug you have noticed and the fix that has been applied in that PR.
Notice that this feature changes the behaviour for email and SMS providers allowing to have multiple active providers set. So try this in a DEV environment to see that it shows the fix but without affecting to your current flow, as a caution. measure.

--•10/23/23, 12:21 PM

Here you have a small reference regarding setting the feature flags in case you need it: https://docs.novu.co/community/run-in-local-machine#set-up-your-environment-variables

Novu

Run Novu in local machine - Novu

Prerequisites and steps to run Novu in local machine. Learn how to set up Novu on your local environment for testing and development.

frnandu•10/23/23, 12:34 PM

Ok, I just tested with IS_MULTI_PROVIDER_CONFIGURATION_ENABLED='true' and the behaviour now is shows FCM as the provider on development (only), and APNs on production (only). And when triggering on production for mixed users, it tries to send for both users using APNs and it fails on the user that has fcm tokens and "providerId": "fcm"

frnandu•10/23/23, 12:36 PM

Do you need more information to help diagnose the problem?

--•10/23/23, 12:49 PM

Yes, can you double check in your development environment that botch FCM and APNs providers are set as active both? We can just double check the behaviour in your DEV environment.
After that I would ask you to double check that you have set the device tokens appropriately for the subscribers, the ones meant to be notified through FCM in that integration and the ones meant to be notified through APNs in that other integration.

frnandu•10/23/23, 12:54 PM

I already checked that few times. Both providers are active. And using REST API curl 'http://api.novu.svc.kube.internal.filmweb.pl:3000/v1/subscribers/<subscriber-id>curl 'http://api.novu.svc.kube.internal.filmweb.pl:3000/v1/subscribers/<subscriber-id> I confirmed that each user to which is triggered the worflow, one has deviceTokens and providerId: "fcm" , and the second one has also deviceTokens and providerId: "apns"

frnandu•10/23/23, 12:55 PM

I confirmed also that when I disabled one of the providers (FCM or APNS) and triggered only for the appropriate user, it sent successfully, so the deviceTokens are confirmed to be correct.

frnandu•10/23/23, 12:59 PM

But on the workflow settings only APNS is visible as active on production.

frnandu•10/23/23, 1:00 PM

frnandu•10/23/23, 1:00 PM

Even tough on you click Configure (or go to Integrations Store) both are active:

frnandu•10/23/23, 1:00 PM

frnandu•10/23/23, 1:03 PM

If I disable APNs, it shows this:

frnandu•10/23/23, 1:03 PM

frnandu•10/23/23, 1:03 PM

Even tough FCM is active on the Integrations Store screen

frnandu•10/23/23, 1:04 PM

And the funny thing is, if you go on Development Tab (because all of this was on production), it shows FCM provider as active, and it just has the same problem but for the APNs...

frnandu•10/23/23, 1:05 PM

--•10/23/23, 1:06 PM

Just last question before asking you to open an issue because indeed it looks we are into something. Confirm me that the menu entry Changes doesn't have any changes to promote from Dev to Production and we are dealing with the same configuration.
Anyway in the Development environment you should already be able to send notifications to both FCM and APNs when both are active.

frnandu•10/23/23, 1:10 PM

In last screen you can see there are no Changes to promote (would show a red number next to it)

frnandu•10/23/23, 1:11 PM

But even in the development (as you can see in last screen), it doesn't seem to have APNs active in the workflow...

Ffrnandu But even in the development (as you can see in last screen), it doesn't seem to ...

--•10/23/23, 1:17 PM

That's true and that's why I asked about changes. It is weird. To just clarify, are you able to enable APNs in your Dev environment?

frnandu•10/23/23, 1:18 PM

In integrations store screen both FCM and APNs providers are active in both Production and Development.

Ffrnandu Click to see attachment

frnandu•10/23/23, 1:19 PM

as you can see here

--•10/23/23, 1:24 PM

Please open a GitHub issue with all this information if you haven't done it yet so we can prioritise the investigation. Thank you very much for being patient and providing so much info.

frnandu•10/23/23, 1:32 PM

But there is already an issue, it got closed but it is not fixed

frnandu•10/23/23, 1:32 PM

https://github.com/novuhq/novu/issues/3797

GitHub

[NV-2935] 🐛 Bug Report: Wrong provider is selected when sending a p...

Description In the case of the push notifications, given that we have 2 active providers in the same organization: fcm (for android and web push notifications) and apns (for ios notifications), w...

Issue while having both FCM and APNS active

Similar Threads

Similar Threads

Similar Threads