situation where the response to an InfluxDB query can be huge (10+ MB easily)
hello I have a situation where the response to an InfluxDB query can be huge (10+ MB easily), which is going into a single list object like so: List<Metric> metrics = await _influxDBService.MachineRead(sql, "metrics"); - this is creating a memory leak and memory fragmentation
please what approach could I take to mitigate this
12 Replies
10MB is not a lot of data and either way will not in itself cause a memory leak or significant memory fragmentation, if any
are you seeing actual performance/memory issues?
yes there is a definite leak related to this call
there are many thousands of objects going into the list
that's not a lot of objects in the grand scheme of things
what is showing you that you have a memory leak and fragmentation?
this is being exposed as an endpoint and if i hit it with a large query to influx, memory usage is going up and not going down
its been difficult to track down exactly, if i look at dumps it doesnt show on the heap
i believe its something to do with LOH fragmentation but i am really a noob on this topic
does memory usage go up continuously until you run out of memory?
in prod this is in a very resource constrained environment so yes
unless you're holding a reference to that list outside the request it's not leaking
the GC will clean it up at some point once it's no longer referenced
unless the influxdb objects allocate unmanaged resources and don't release them that single line of code doesn't look like any kind of leak to me
would you by any chance be familiar with the influxdb3 csharp nuget package?
no, but if you use a profiler like dotMemory it has a memory snapshotting feature that can show you a summary of allocated objects and their count/memory usage
yeah it all looks good i have looked at it
it seems to be native memory
i would guess it's an issue with the library, unless there's something special about
Metricsits a very disgustingly large object
thanks for the help anyhow it was reassuring