Microsoft sync works, only it doesn't... (multi-tenant)
Hi all and thanks for a great solution!
I'm self-hosting on Twenty on an internal server, and finally got the MS integrations going with SSL.
Logging in with MS works like a charm, and connecting my account also works - except that nothing actually gets imported.
It is stuck on the 'Importing' status, and I can't for the life of me find anything that could indicate what is wrong in any logs.
All cron jobs are started as well.
All ideas on how to resolve this would be greatly appreciated
44 Replies
did you pull the latest version from the main branch ? We fixed a lot of issues on this topic this week
Hi @Guillaume, thanks for the quick response!
The install was done on Saturday by the one-click Docker install script, so unsure if it is the latest version or not.
Hey @El Odve, could you please share the TAG value of your one-click install
.env
Or execute echo APP_VERSION in your twenty-server containerGuess I'm running 0.52.6
it would be great to try on version 52.8
we fixed a couple of issues between 52.7 and 52.8
Ok, will try. I see now that my worker is down as well, will try to figure out why.
The Best Way(tm) to upgrade is to run the one-click installer again?
@El Odve to upgrade you should update the TAG value in your
.env and restart your containersThanks @Prastoin, just figured it out and did that. Up and running with 0.52.8 now, but seems like the issue is still there. According to the health panel the worker is down, but the system works and I can't find any errors in the logs...
When restarting the containers, the twenty-server-1 is reported as unhealthy, but I can still access everything as normal and I don't see any error in the logs.
That's still a step further !
Could you please share your worker's logs ?
Sure, just tell me what I should to to provide you with the exact log you want!
I'm fairly new to docker environments...
Also, it seems like the worker never actually starts, as it depends on the server and that is reported as unhealthy when starting - despite actually running...
Maybe worth mentioning that it's running using a self-signed SSL certificate, which was a bit of a hassle as the certificate and key had to be manually copied to the server container and I had to turn off ssl in the postgres env string. So possibly something related to that...
I have a test version running locally on my computer without SSL, just using localhost as the server URL - this instance is reported as healthy including the worker, but here I the sync fails with the message 'Sync lost with mailbox...'
Ok, so I'm getting somewhere here. My main issue is related to SSL and the self-signed CA in combination with the health test in the docker compose file, which tests at http://localhost:3000/healthz. Even though the paths to the certificate and key are specified in the config, they are not copied to the docker image and I have to do that manually. Then the next issue is that the curl command to test is a) using the wrong server address (I have another URL that is also specified in the config) and b) that it will fail anyway because it doesn't like self-signed CAs. After editing the test with the correct URL and adding --insecure to allow for a self-signed CA, the worker is now at least starting.
All cron jobs are now also started from the worker, so fingers crossed...
@Prastoin: Finally an actual error for you to look at:
ERROR [CalendarFetchEventsService] Error: Access token is undefined or empty. Please provide a valid token.For more help - https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/docs/CustomAuthenticationProvider.md
GitHub
msgraph-sdk-javascript/docs/CustomAuthenticationProvider.md at dev ...
Microsoft Graph client library for JavaScript. Contribute to microsoftgraph/msgraph-sdk-javascript development by creating an account on GitHub.
at parseMicrosoftCalendarError (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/drivers/microsoft-calendar/utils/parse-microsoft-calendar-error.util.js:31:20)
worker-1 | at MicrosoftCalendarGetEventsService.getCalendarEvents (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/drivers/microsoft-calendar/services/microsoft-calendar-get-events.service.js:42:84)
worker-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
worker-1 | at async CalendarFetchEventsService.fetchCalendarEvents (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/services/calendar-fetch-events.service.js:43:47)
worker-1 | at async CalendarEventListFetchJob.handle (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/jobs/calendar-event-list-fetch.job.js:53:17)
worker-1 | at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:111:17)
worker-1 | at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:103:13)
worker-1 | at async /app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:79:17
worker-1 | at async Worker.processFn (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/drivers/bullmq.driver.js:41:13)
worker-1 | at async /app/node_modules/bullmq/dist/cjs/classes/worker.js:520:32 {
...and after disconnecting and reconnecting the account, we're now back to the 'Sync lost' error:
Which seems to be caused by the above error.
Note that logging in using the Microsoft connector works like a charm.
Note that microsoft requires a paid account in order to have the sync possible. is
[email protected] a free account ?No, it's a corporate account.
so a 365 paid account right ?
that's good
That's right.
I've set up all the required permissions in azure, the secret is valid, application ID is fine and redirect URIs work.
To me, you have an issue with your setup that is related to this part of the code where we create the connection to microsoft graph API
In my opinion, the refresh token that you can look in the database is not good. I don' t know ehy but maybe you can copy it and test it in Postman or another tool.
to find your token, go to your db and look for your connected account, then look at the "refreshToken" related to your account
How/where do I do that?
@Guillaume: I figured out how to test this using Postman, and the refreshToken as copied from the database is valid. Something else is amiss. Could you point me to where in the source code you create the connection? With a bit of luck I might be able to debug something better then.,..
Sure
You can start by having a look at this file
/packages/twenty-server/src/modules/connected-account/oauth2-client-manager/drivers/microsoft/microsoft-oauth2-client-manager.service.ts
the getOAuth2Client method is the one we use then to instanciate the connections for messaging and calendarThanks! I really want to get to the bottom of this. 🙂 Loving Twenty CRM so far, just need to get this bit sorted and it will be perfect for our purposes!
Sure !
Here is a fresh error log from worker-1 upon failed import. This crashed the Connected Accounts service, which then takes a few minutes to get back up.
Exception Captured
undefined
[
'Unknown error importing messages for message channel 5f45fdcb-a6d6-4f32-9be8-8a426d5c18bd in workspace 2fa67d60-5c17-4a6e-becd-2e70f4434d81: Request failed with status code 401'
]
Exception Captured
undefined
[
MessageImportException [Error]: Request failed with status code 401
at MessageImportExceptionHandlerService.handleUnknownException (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/services/messaging-import-exception-handler.service.js:110:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async MessageImportExceptionHandlerService.handleDriverException (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/services/messaging-import-exception-handler.service.js:62:17)
at async MessagingMessageListFetchJob.handle (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/jobs/messaging-message-list-fetch.job.js:128:13)
at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:111:17)
at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:103:13)
at async /app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:79:17
at async Worker.processFn (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/drivers/bullmq.driver.js:41:13)
at async /app/node_modules/bullmq/dist/cjs/classes/worker.js:520:32
at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:747:24) {
code: 'UNKNOWN'
}
]
These are the API permissions set up in Azure. Note that signing in works.
still an unauthorized error. Looks like there is. an issue in your token creation / azure permission
I tested the refreshToken using Postman yesterday, and the token itself seemed fine.
Could there be some security setting company wide in Azure that breaks this?
Not sure at this level sorry. I am quite noob on Azure specification
...that makes two of us...
haah
Me with Microsoft doc lol
good luck for your research
let me know if you find something
😆 Will do!
BTW, do you keep the response body from the Microsoft Graph API in the logs somewhere so I can look there? If not, I'll try to replicate the call that fails once I locate it in the source code.
nope, you need to add some logs
I think I'm getting to the root of this now. The accessToken field in the database is blank. Am I correct in assuming that you use the common access point in OAUTH? I haven't seen any place I can specify the tenant id, so I think that is a safe assumption? If I try to obtain an accessToken manually using curl and the common tenant then this is indeed currently blocked by conditional access. If I try using our tenant ID, I can get the access token without any issues.
Do you have any plans to support single tenant applications?
So essentialy you're right, it is an unauthorized error. For me the simplest workaround would be to be able to set a TENANT_ID variable in the config which has the default value common
ah yes good point. we only support multi tenant ! there is. a thread somewhere in discord talking about it but it is not on our roadmap sorry.
You can still use twenty as multi tenant or make a clone to change this line if it's a mandatory feature right now for you !
I updated the title for other users to find out 🙂
For me it would probably be easiest to make a clone, I think. At least there I now know what I need to do. 🙂 Fixing the permissions in Azure is more of an unknown at this point... 😉 Is there more magic to supporting single tenant than having a MICROSOFT_TENANT_ID env variable that is set to 'common' by default and using this value in the url used to fetch the access tokens? I see it is used in two files, packages/twenty-server/src/modules/connected-account/oauth2-client-manager/drivers/microsoft/microsoft-oauth2-client-manager.service.ts and packages/twenty-server/src/modules/connected-account/refresh-tokens-manager/drivers/microsoft/services/microsoft-api-refresh-tokens.service.ts
i think you got it right
nothing more
Hmmm... There's something else going on as well. I know get an error when importing calendar events, which claims the access token is undefined or empty. When replicating the relevant call from the code manually using the values from the connectedAccount table in the database (which is now successfully populated), I get a valid response including a new accessToken.
Exception Captured
{ workspace: { id: '2fa67d60-5c17-4a6e-becd-2e70f4434d81' } }
[
CalendarEventImportDriverException [Error]: Access token is undefined or empty. Please provide a valid token.For more help - https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/docs/CustomAuthenticationProvider.md
at parseMicrosoftCalendarError (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/drivers/microsoft-calendar/utils/parse-microsoft-calendar-error.util.js:31:20)
at MicrosoftCalendarGetEventsService.getCalendarEvents (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/drivers/microsoft-calendar/services/microsoft-calendar-get-events.service.js:42:84)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async CalendarFetchEventsService.fetchCalendarEvents (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/services/calendar-fetch-events.service.js:43:47)
at async CalendarEventListFetchJob.handle (/app/packages/twenty-server/dist/src/modules/calendar/calendar-event-import-manager/jobs/calendar-event-list-fetch.job.js:53:17)
at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:111:17)
at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:103:13)
at async /app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:79:17
at async Worker.processFn (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/drivers/bullmq.driver.js:41:13)
at async /app/node_modules/bullmq/dist/cjs/classes/worker.js:520:32 {
code: 'UNKNOWN'
}
]
GitHub
msgraph-sdk-javascript/docs/CustomAuthenticationProvider.md at dev ...
Microsoft Graph client library for JavaScript. Contribute to microsoftgraph/msgraph-sdk-javascript development by creating an account on GitHub.
@Guillaume: Don't know why, but rebuilt with 0.53.2, patched the two files in question within the container and restarted and NOW it works. 😄
❤️
Well done @El Odve ! Enjoy coding importing !
Thanks! The least amount of coding I've done to get something working... 😄