C#•3y ago

❔ EFCore Sqlite Async: RAM and CPU Problem

So my code does the following:

30 Targets are being processed at once via async operations. Each second for each target I create 20 new Children of the target. So per second I have 20 * 30 new children. Each 20 children I call the "SaveAsync" method.

However before I add a child to the database I want to ensure that the child does not exist yet, if it exists already I update its properties.

var existingUser = await ctx
                        .Children
                        .FirstOrDefaultAsync(x => x.RestID.Equals(scrapedUser.RestID));

var existingUser = await ctx
                        .Children
                        .FirstOrDefaultAsync(x => x.RestID.Equals(scrapedUser.RestID));

var existingUser = await ctx
                        .Children
                        .FirstOrDefaultAsync(x => x.RestID.Equals(scrapedUser.RestID));

var existingUser = await ctx
                        .Children
                        .FirstOrDefaultAsync(x => x.RestID.Equals(scrapedUser.RestID));

Im doing it with this code. I am assuming that EFCore loads all the children objects into memory, that leads to very high memory exposure, how can I prevent this? It should operate on Database level each time and not load all objects in RAM.
It is important because I have millions on child object.

Second issue is the CPU issue. Without doing the database actions, such as SaveAsync, FirstOrDefaultAsync the CPU is around 1% (normal for my Task), however with those methods after around 2mins I have a CPU over 90%.

Angius•10/8/23, 1:07 PM

No, EF does not load the whole database into memory

Angius•10/8/23, 1:07 PM

It generates an SQL query with a WHEREWHERE and LIMIT 1LIMIT 1 in this case

KroksOP•10/8/23, 1:07 PM

Hmmm whats the issue with the memory leaks then? Without DB usage I have consistently around 400

KroksOP•10/8/23, 1:07 PM

and with it just goes up and up

KroksOP•10/8/23, 1:08 PM

I can send all operations that I do on Database side

KroksOP•10/8/23, 1:08 PM

Well basically I only do these two that I just said

UltraWelfare•10/8/23, 1:08 PM

Why not try profiling it to see which objects are doing the damage

Angius•10/8/23, 1:08 PM

Huh, well, the only way I can see it being an issue is if maybe EF cannot translate that .Equals().Equals()

Angius•10/8/23, 1:08 PM

You should just take a look at what query gets generated

KroksOP•10/8/23, 1:10 PM

   var existingUser = await ctx
                        .Children
                        .FirstOrDefaultAsync(x => x.RestID.Equals(scrapedUser.RestID));

                    if (existingUser != null)
                    {
                        existingUser.ScrapedDate = scrapedUser.ScrapedDate;
                        existingUser.Followers = scrapedUser.Followers;
                        existingUser.CanDM = scrapedUser.CanDM;
                        existingUser.StatusesCount = scrapedUser.StatusesCount;
                    }
                    else
                    {
                        ctx.Children.Add(scrapedUser);
                    }

   var existingUser = await ctx
                        .Children
                        .FirstOrDefaultAsync(x => x.RestID.Equals(scrapedUser.RestID));

                    if (existingUser != null)
                    {
                        existingUser.ScrapedDate = scrapedUser.ScrapedDate;
                        existingUser.Followers = scrapedUser.Followers;
                        existingUser.CanDM = scrapedUser.CanDM;
                        existingUser.StatusesCount = scrapedUser.StatusesCount;
                    }
                    else
                    {
                        ctx.Children.Add(scrapedUser);
                    }

so this code looks fine at first glance?

Angius•10/8/23, 1:10 PM

Yep

KroksOP•10/8/23, 1:11 PM

so the DBSet "Children" is not loaded into memory?

Angius•10/8/23, 1:11 PM

Could probably use ExecuteUpdateAsync()ExecuteUpdateAsync() instead, but this is valid

Angius•10/8/23, 1:11 PM

Nope, it should not be loaded unless you explicitly load it all

KroksOP•10/8/23, 1:12 PM

I only do that + SaveChangesAsync after each 20 new children that I add to the DBSet

KroksOP•10/8/23, 1:12 PM

nothing else

KroksOP•10/8/23, 1:12 PM

so whats my best approach to solve this ?

Angius•10/8/23, 1:13 PM

Well, at a glance, there's... nothing to solve, everything seems fine

Angius•10/8/23, 1:13 PM

Unless you're doing something really stupid like newnewing up a new DbContext with every loop

AAngius Unless you're doing something really stupid like `new`ing up a new DbContext wit...

KroksOP•10/8/23, 1:14 PM

I dont actually, I have 30 DbContexts as otherwise I get an exception for concurrency.
Per async Task that generates new childs per target one BotContext

Angius•10/8/23, 1:14 PM

How do you get those contexts?

AAngius How do you get those contexts?

KroksOP•10/8/23, 1:15 PM

using var ctx = new BotContext(_db);

using var ctx = new BotContext(_db);

KroksOP•10/8/23, 1:15 PM

or what do you mean?

Angius•10/8/23, 1:15 PM

Yeah, that seems fine-ish

KroksOP•10/8/23, 1:15 PM

_db is path to sqlite

Angius•10/8/23, 1:15 PM

I thought you might be not disposing of them, hence the memory issues

Angius•10/8/23, 1:15 PM

Huh

KroksOP•10/8/23, 1:15 PM

I mean they stay alive, these tasks that I was speaking of do not exit for hours

KroksOP•10/8/23, 1:15 PM

so this instance stays alive for hours

KroksOP•10/8/23, 1:16 PM

30 instances

Angius•10/8/23, 1:16 PM

I'm out of ideas, I'm afraid

Angius•10/8/23, 1:16 PM

Try asking in #database maybe? Link this thread there, provide a short summary

AAngius Try asking in #database maybe? Link this thread there, provide a short summary

KroksOP•10/8/23, 1:16 PM

Ok thanks for trying to help.

KKroks so this instance stays alive for hours

Insire•10/8/23, 1:22 PM

if you dont dispose the dbcontext then each added Child/User will continue to be tracked. meaning memory usage will increase

Insire•10/8/23, 1:23 PM

same for the updated user, since you seem to be tracking entities by default

Insire•10/8/23, 1:24 PM

so its possible, with your current approach, to load the entire database into memory multiple times (as many times as you have semi persistent dbcontext instances)

Insire•10/8/23, 1:25 PM

the solution is to dispose the dbcontext earlier

IInsire if you dont dispose the dbcontext then each added Child/User will continue to be...

KroksOP•10/8/23, 1:25 PM

makes sense

KroksOP•10/8/23, 1:25 PM

I will defo try that

KroksOP•10/8/23, 1:26 PM

what about the CPU usage? Might it be because I save too frequently?

KroksOP•10/8/23, 1:26 PM

30 Tasks save it each 20 new children

KroksOP•10/8/23, 1:26 PM

so basically each second it saves 30 times

KroksOP•10/8/23, 1:26 PM

or isnt that a problem

Insire•10/8/23, 1:27 PM

saving and writing to database in itself isnt cpu intentive, its IO intensive

Insire•10/8/23, 1:27 PM

but your queries can become expensive, given enough data to process

Insire•10/8/23, 1:28 PM

e.g. by having no or outdated indezes

IInsire if you dont dispose the dbcontext then each added Child/User will continue to be...

KroksOP•10/8/23, 1:28 PM

but then I guess I have the issue that I create them too frequently. Like I would have a new context per task each second, what is like suggested?

Insire•10/8/23, 1:28 PM

dbcontext is pretty cheap to create

IInsire e.g. by having no or outdated indezes

UltraWelfare•10/8/23, 1:28 PM

Or a lot of indexes...

IInsire dbcontext is pretty cheap to create

KroksOP•10/8/23, 1:29 PM

so its fine if I have it in a loop but dispose after each iteration

❔ EFCore Sqlite Async: RAM and CPU Problem

Similar Threads

Similar Threads

Similar Threads