IMAP Account Sync 100% CPU LOAD
Hey,
I already posted in #š”ļø±general , but I was now able reproduce the problem.
1. Add a new IMAP with around 500+ Emails
2. check cpu load
3. Twenty will become unresponsive
I currently use a VPS with 4GB RAM and 2vCPUs that 2 steps are able to make the vps completely unresponsive (no ssh login possible, twenty UI unresponsive, etc..) After an minute or two the server recovers or restarts because out of memory.
With a smaller Inbox (50-100Mails) itās working and the synced status appear nearly instant.
Is this a known Issue?
96 Replies
Edit: the import process is also taking forever for even small inbox (500 mails). I started that process again while posting that thread and itās still running. Is that normal?
Update: Now itās finalized but Sync is not showing all results
For example x.yyy@z.com not working.
X is the first letter of first name
YYY is the last name
Z is the domain
XX@domain.com is also not every time appearing. Some yes but not all.
Assuming my domain my domain is Google.com all my internal emails arenāt visible. Is that the way it should work?
I did now a comparision. I would assume only 70% are synced (People & Companies) but from those only 30% of the Emails are visible.
I just created a GH Issue: https://github.com/twentyhq/twenty/issues/14520
GitHub
IMAP causes heavy resource load Ā· Issue #14520 Ā· twentyhq/twenty
Bug Description If you change some IMAP Settings which triggers the Importing Logic, my VPS will become unresponsive. Which shouldn't happen for an imap import. Currently it uses 100% of the CP...
Hey @DerPicknicker,
Twenty will become unresponsiveThe whole app ( server client and worker ) or just worker ? I assume you're using the one click install ? which means you're deploying the three of them on the same pods ( which would make sense for global outage if worker get overloaded )
Hey @prastoin ,
yes on same host.
I did now a comparision. I would assume only 70% are synced (People & Companies) but from those only 30% of the Emails are visible.I don't really know email synchro implem details recover on failed chunk maybe you're missing the one that failed due to cpu issues
Maybe thats a coincedence.
From my understanding if it's the very for time your're importing your emails you could upgrade your cpu just for the duration of the initial import
If you need to be able to handle same amount of email import volume then you will need more cpu
In the other hand you could try deploying your worker on its own
For me its weird because even on a low powered device those inboxes arent big
500 mails is like a small one
So if it comes to fail it doesnot impact the twenty-server that will I think handle recovery of the failed import
Yes for sure
but
Indeed that's pretty light
This doesnt make sense to me
Could you please confirm what twenty version you're using ?
1.5.3
Any error in worker logs ?
I guess heap memory
But if anything else might be helpful
I use an low power device for my HomeAssistant server that handles a music library and 10 streams at same time with 20% CPU Load. So for me there must be something wrong š
Should I trigger Re-Sync again and post docker compose logs?
Are you sure you allowed the correct cpu amount you're thinking of to the twenty instance ?
Yep always a plus
I have 2Vcpus on my VPS instance
Twenty is currently importing emails in backround (same inbox which I currently face this issue)
Ok cool
That's some neat graph haha
Ok so indeed there're might be an issue
Currently Using 1,9gb RAM directly after starting going up to 3,9gb (4gb is max) so this is also a bit problematic.
Unless I'm mistaken there's a new feature in the lab allowing a specific folder import
If possible I would like you to try importing 500 times a basic email
You mean just simple Email without Responding / threads?
Yep
Hetzner hosting š
I'd like to attest that this is not the volumes of 500 but the integrity of the initialy imported 500 emails that are causing the issue
Do you have an idea why itās taking ages to finish the import ? Because after the spikes to 100% cpu load itās taking hours to finalize
Give me a second let me try
Take all your time no worries
Hours not normal indeed
It took like 4hours yesterday
After the spike
@prastoin created now an inbox with 28 elements with attachments, blank emails, html ones. Let me force to update again.
After changing the credentials and directly turning off the inbox with 500 elements its still utilizing all cpu power
Is top graph value 100% ?
yes
And I found another bug
Could you please remove any attachements from the imported emails ?
Please feel free to share
Its hard because the Inbox-selector always defaults back to the sent / inbox setting. So I always trigger a full importing which causes my server to be unresponsive.
This is super annoying because I cannot test your needed things properbly.
Also funny this is the ssd bandwith while importing.
Now one is successfully imported (<100 Mails) the other one is still importing (around 500) this seems fine.
@prastoin
Here is my folder setting at the moment.
After changing the credentials to trigger re-sync it falls back to sent and inbox.
@prastoin the IMAP import seems to be partially broken. Because after setting only my test folder for reading emails are showing up which werenāt visible but not if I read my whole inbox in.
cc @NEO, will have more information than I regarding in wip IMAP integration
Thank you for all your support till here
@DerPicknicker Hi
What provider are you using for email?
All-Inkl
Its a german provider
@NEO
This provider works seamlessly on Windows and iDevices via IMAP.
Could you summarize the issue(s) so far you've had? just to confirm
Big Inbox (500 Mails) uses 1-3mins all CPU Power until nothing works anymore (I guess even unitl out of memory) small one (<100) works fine. I think this also causes that not all Emails are synced. And the Sync taking ages (4-5 Hours for the 500 Mail inbox).
From my 500 Mails not all Contacts are created as person elements. And if created not all Emails are linked to that person. Even if the Frontend says its synced. Not all Emails are available from that Person.
This HIGH cpu usage is like an ddos attack to my server even SSH is not working because its fully used by twenty (LOL).
@NEO here is the graph for cpu usage
Interesting, will investigate and update you soon
Thank you for your support. If you need something please let me knwo.
Maybe you could also check this. Maybe its related..
Your RAM allocation was 2GB right?
and 2vCPUs?
4GB and 2VCPU
I see
the whole Server is 4GB
Currently 2/4GB used. So quite relaxed.
This should be fixable I think, will update you
Though (in my opinion) it's overkill to change credentials everyday if you use a strong generated password
thank you. Twenty is just awesome for me.. I am a Startup Company of 2 Guys and its blown my mind. Thank you for fixing...
I am not changing everyday. From time to time. But the biggest problem is that saving credentials is the only way to force twenty to update the IMAP Folders. It would be better on the folder page a small refresh icon to get latest IMAP folders before syncing. And after making a change in the selected folders start the actual sync.
folders are auto synced when new messages are imported
the process is inter linked
If I create a new Folder. I need to update the credentials without making a change that only makes them visible.. For me. I created, opened twenty and they arent visible until I click on connection settings, save. then the newly created folders will appear.
That's odd, but I should also mention new messages are imported periodically every 2 minutes
so you wouldn't see a new folder instantly
I never waited 2 mins. Maybe they will appear but for me it would be more intuitive to have a small refresh icon for only the folder page.
I understand your perspective, will discuss it with Felix and get back to you
No problem take your time. Twenty is already awesome. I donāt want to criticize at all. Just testing currently a lot š
Feedback is always welcome whether positive or negative
Just let me know if you need some more information.
Will do!
how can I create a trigger that match with specific value of
Stage ? For example I only want to run the workflow when value of Stage is Meeting
Hey @longbuibao, unless I'm mistaken this un-related to this help thread, please create a dedicated one š
@prastoin @NEO
I just send now an Email to an Person which wasnt created via the IMAP Sync (I created that person). Now I copied the email into the Folder which I created here for testing purposes and that Email is never linked to the person (waited 5-8mins).
Hi, please donāt tag Paul Iām responsible for IMAP issues
Sorry wasnt aware which one is responsible.
Iām currently occupied with some other issue, but will investigate yours as soon as possible
No problem I donāt want to stress you in any way. Just playing around.
@DerPicknicker Quick update on syncing large mailbox ( > 500) I tested it locally and it synced fine, haven't profiled CPU and RAM usage yet it's tricky on macOS I'll need to setup linux VM
But also with threads?
Or plain emails?
Nearly 90% are email threads
Just limit ram in docker engine
And you could also limit cpu cores
I guess you could with some finnicking with volume mounts, I run bare metal for development
Yes
How big is your average email? is it plain text? HTML?
inline content/attachment?
80% HTML
Attachment sizes varies between 1-30MB
Mostly as attachments.
Would it be possible to get some sample emails? you could DM me the info and of course redact any personal identifiers
We could also do a short call and I show you the email inbox and we can stress the import and Check logs together
Or I create an inbox at my provider and you do your testing again
a
.eml export for a few emails would sufficeBecause I donāt see all emails too. So somewhere is something wrong with;)
Is your cron job setup correctly?
sometimes that's the issue
just narrowing down the possibilities
How to check?
I face 3 Issues:
- Copy email into a folder watched by twenty wonāt show up
- After synced with extensive resource consumption not all people are visible
- After manually creating a user emails will never get matched to that person
I have the feeling that this is caused by the usage consumption bug
Exclude non-professional emails is enabled by default
The ones that didn't show did they happen to be gmail/outlook?
Nope custom domain
Okay let's try addressing the spike first
Anything on this?
I am using not outlook. Give me some time to figure out
If you're macOS and use Mail.app you can drag the emails to desktop it will export them
I will do it in 2mins how many do you need?
I think 4-5 should be enough
Okay give me a second