C#C
C#•2y ago
Jesse

Handling reading large files in C#

How do I go about reducing memory usage to handle large files in x86 C#? My current code is as follows:
                FileStream fs = new FileStream(filepath, FileMode.Open, FileAccess.Read);
                using (BinaryReader br = new BinaryReader(fs))
                {
                    byte[] bytes = new byte[0];
                    using (MemoryStream test = new MemoryStream())
                    {
                        fs.CopyTo(test);
                        bytes = test.ToArray();
                    }

                    byte[] searchBytes = Encoding.UTF8.GetBytes("test");
                    List<long> positions = new List<long>();

                    foreach(long pos in Extensions.SearchStringInBytes(bytes, searchBytes))
                    {
                        positions.Add(pos - 4);
                    }
                }

When reading a large file (>500MB) the memory usage skyrockets to 2GB.
The result is that it only works in x64 build, as x86 results in a OutOfMemoryException near 1GB memory usage.
I have thought of reading the file in "chunks" but I'm not sure how. Any other suggestions aside from making the program x64 only?
Was this page helpful?