Twenty•2mo ago

Syncing google emails does not work

Hey there, I tried to setup twenty, and the gmail integration for my company and I run into this error : https://github.com/twentyhq/twenty/issues/15241 Has anyone faced that before ? I'll investigate - it seems that error handling for gmail is not working fine, i'll check - but it takes time to setup as I need to set up dev env on the vps machine, so if anyone has clue I'm taking them Thanks a lot for your help guys, product is awesome tho

GitHub

Impossible to sync Google Messaging · Issue #15241 · twentyhq/twenty

Bug Description So I set up a twenty crm app on a vps with https under nginx proxy. Everything works well. I added following https://twenty.com/developers/section/self-hosting/setup#gmail-&-goo...

21 Replies

Prastoin•2mo ago

Hey @Dr00ps I assume you're AntoineXev ? If you're facing the very same issue in local than your deployed instance there's very high chance that's it's due to a configuration missmatch @Guillaume does this ring a bell to you ? I'm not very familiar very this configuration

Dr00psOP•2mo ago

Yep that's me sorry ! Well maybe, but that seems strange - the app is able to retrieve emails - some at least (I saw them) and and they're well parsed by the system. Somehow some gmail api calls seems to return 500 or so - making the whole batch fails and importing nothing But I might be missing something. Here is the variables I have

PG_DATABASE_URL=postgres://postgres:xxxxx@xxxxx:5432/default
REDIS_URL=redis://xxxx:6379

APP_SECRET=xxxxx
EXCEPTION_HANDLER_DRIVER=CONSOLE
SENTRY_DSN=https://ba869cb8fd72d5faeb6643560939cee0@o4505516959793152.ingest.sentry.io/4506660900306944
MUTATION_MAXIMUM_RECORD_AFFECTED=100

FRONTEND_URL=http://localhost:3001
MESSAGING_PROVIDER_GMAIL_ENABLED=true
CALENDAR_PROVIDER_GOOGLE_ENABLED=true
AUTH_GOOGLE_CLIENT_ID=xxxx.apps.googleusercontent.com
AUTH_GOOGLE_CLIENT_SECRET=xxxxxx
AUTH_GOOGLE_CALLBACK_URL=http://localhost:3000/auth/google/redirect
AUTH_GOOGLE_APIS_CALLBACK_URL=http://localhost:3000/auth/google-apis/get-access-token

PG_DATABASE_URL=postgres://postgres:xxxxx@xxxxx:5432/default
REDIS_URL=redis://xxxx:6379

APP_SECRET=xxxxx
EXCEPTION_HANDLER_DRIVER=CONSOLE
SENTRY_DSN=https://ba869cb8fd72d5faeb6643560939cee0@o4505516959793152.ingest.sentry.io/4506660900306944
MUTATION_MAXIMUM_RECORD_AFFECTED=100

FRONTEND_URL=http://localhost:3001
MESSAGING_PROVIDER_GMAIL_ENABLED=true
CALENDAR_PROVIDER_GOOGLE_ENABLED=true
AUTH_GOOGLE_CLIENT_ID=xxxx.apps.googleusercontent.com
AUTH_GOOGLE_CLIENT_SECRET=xxxxxx
AUTH_GOOGLE_CALLBACK_URL=http://localhost:3000/auth/google/redirect
AUTH_GOOGLE_APIS_CALLBACK_URL=http://localhost:3000/auth/google-apis/get-access-token

Thanks for the help @prastoin

Guillaume•2mo ago

Env vars sounds ok to me. I would focus on the error message i see in yout github issue

worker-1  |     MessageImportException [Error]: Temporary error occurred multiple times while importing messages for message channel f8511ee6-7ce4-4291-ba4e-4c314bd801f5 in workspace 91a76ba5-845c-4336-848b-9ccb7663db3c: Internal error encountered. for message with externalId: 18378e3d2240b4e2

Dr00psOP•2mo ago

+ messages lists are well fetched

Guillaume•2mo ago

sounds like maybe there is one message with special char or very heavy ? it would be nice if you could identify which email relates to this id: 18378e3d2240b4e2

Prastoin•2mo ago

Trying to link another google account might help for the dichotomy

Dr00psOP•2mo ago

Yep right it does seems like something similar I'll try with another also It's actually a Jira confluence report ahah

Guillaume•2mo ago

interesting is it big ? is it in a special alpahbet ? strange emoji ? could be anything it can also come from the consluence sender email, title, or even cc bcc... anything strange you spotted ?

Dr00psOP•2mo ago

There's emoji in it - alphabet seems ok, and not so big I'm investigating all the failed emails see if they have something in common Ok, @Guillaume It's definitly an issue with the gmail batch api - but also with the app. I used a proxy to reverse eng the calls to gmail. Batch calls to gmail are all ok -> returns 200 But inside, some of the mails calls return 500. I'll investigate on my side why the api returns 500 for those (they seems to be weirdly random). But the imo the app should not fail the whole import / batch if only one mail is in 500. Do you want me to propose a fix afterwards ?

--batch_spbVVedGxrpI3XHV9ZfLRLurNuxyg5fs
Content-Type: application/http
Content-ID: response-

HTTP/1.1 500 Internal Server Error
Vary: Origin
Vary: X-Origin
Vary: Referer
Content-Type: application/json; charset=UTF-8

{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "errors": [
      {
        "message": "Internal error encountered.",
        "domain": "global",
        "reason": "backendError"
      }
    ],
    "status": "INTERNAL"
  }
}

--batch_spbVVedGxrpI3XHV9ZfLRLurNuxyg5fs
Content-Type: application/http
Content-ID: response-

HTTP/1.1 500 Internal Server Error
Vary: Origin
Vary: X-Origin
Vary: Referer
Content-Type: application/json; charset=UTF-8

{
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "errors": [
      {
        "message": "Internal error encountered.",
        "domain": "global",
        "reason": "backendError"
      }
    ],
    "status": "INTERNAL"
  }
}

Guillaume•2mo ago

The thing is that is should not fail at all 🙂 we aim towards 100% success rate on email imports. I think we already try to catch each email error to import the rest of the email. If you find a fix though it would be appreciated

Dr00psOP•2mo ago

Sure it's annoying. I'll try to check how we can fix that during the week. I'll also isnpect the gmail batch api

Dr00psOP•2mo ago

Thanks for the great help guys. I found out a few things. With the current code, exception should be handled. However, with my setup, idk why (maybe an env variable that is not well set ?) a throw in a worker seems to stop this work from ongoing. It came from here: https://github.com/twentyhq/twenty/blob/main/packages/twenty-server/src/modules/messaging/message-import-manager/drivers/gmail/services/gmail-handle-error.service.ts It throw the gmail error at the end of the day - stopping the working. I think your approach was to continue with those errors: https://github.com/twentyhq/twenty/blob/main/packages/twenty-server/src/modules/messaging/message-import-manager/drivers/gmail/services/gmail-get-messages.service.ts My logger is console, but maybe I need to configure some other env var so those throw are not blocking the processes ? Removing the throw seems to fix it also. Anyway, trying to recatch the faulty message seems to fix it, and having a no failure in fetching the email, dispite the frequent 500 errors from the gmail api (those seemed frequent). My approach was to store those failed message, and relaunch the fetch of those until the list of error size is null or the same size as before (unfetchable emails). Do you want me to open a pr - idk if my code is clean enough for you ? if yes, do you have a branch naming that I should respect ?

GitHub

twenty/packages/twenty-server/src/modules/messaging/message-import-...

Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty

GitHub

twenty/packages/twenty-server/src/modules/messaging/message-import-...

Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty

Guillaume•2mo ago

you can PR. AntoineXev-fix-gmail-sync-issue will be good enough as branch name

William Lee•2mo ago

I think I am having same issue but I am not as technical to understand it keeps telling me to reconnect my email and whenever I modify mail or calendar settings it says error. However it did at one point import all my contacts which I wish it didn't

Arthur•2mo ago

Yeah this is a bug since 1.7.x same for microsoft mail sync in general just isn't ready/stable yet

Dr00psOP•2mo ago

@Arthur @William Lee it's a different bug - I have it also - I think token isn't renewed. I'm trying to see how it can be fixed

Prastoin•2mo ago

Hey there google token renewal fixes are going to be released tomorrow in 1.10

Karel Navratil 16093•2mo ago

Does it fix also Microsoft issue?

Prastoin•4w ago

Hey guys should be fixed in latest 1.10.2

Prastoin•4w ago

Please follow upgrade guide global guidelines https://docs.twenty.com/developers/self-hosting/upgrade-guide#general-guidelines