❔ Whats the fastest way to write this data to a file? (C# Console App.)

WaffleDevs · 2023-08-17T01:19:50.440Z

I need to write an int[][] to disk. Each int[] has 2 values, and the full int[][] has 100mil entries. What would be the fastest method of writing this to disk?

A

Angius•8/17/23, 1:21 AM

StreamWriter

StreamWriter

for sure, this amount of data should be streamed

A

Angius•8/17/23, 1:22 AM

Perhaps there are some better, lower-level ways, but I'm unaware of them

J

Jimmacle•8/17/23, 1:23 AM

if each int[] has 2 values can it be a tuple instead? then you could at least get it all in one contiguous block of memory, though IO will probably be the bottleneck regardless

W

WaffleDevsOP•8/17/23, 1:23 AM

Shall i just combine them like such?

sw.WriteLine(xY[0] + ", " + xY[1]);

sw.WriteLine(xY[0] + ", " + xY[1]);

J

Jimmacle•8/17/23, 1:24 AM

if you want this to be fast you shouldn't even be touching strings

W

WaffleDevsOP•8/17/23, 1:24 AM

I thought so from testing

, i just dont know any other way to combine them with a seperator.

J

Jimmacle•8/17/23, 1:24 AM

the format is simple and consistent so you could just dump the bytes out into a file directly

J

Jimmacle•8/17/23, 1:25 AM

well, you didn't specify any output format requirements so that will affect what solutions you can use

W

WaffleDevsOP•8/17/23, 1:26 AM

Ah my bad

.
I would prefer that each int[] is written on seperate lines, and each value is seperated with at least 1 non number character.

W

WaffleDevsOP•8/17/23, 1:26 AM

Ex:

1,2
3,4
5,6

1,2
3,4
5,6

J

Jimmacle•8/17/23, 1:27 AM

then just get rid of the string concatenation and write each part of the line individually with streamwriter calls

W

WaffleDevsOP•8/17/23, 1:28 AM

so

using (StreamWriter sw = new StreamWriter("CDriveDirs.txt"))
{
    foreach (int[] xY in worlds)
    {
        sw.WriteLine(xY);
    }
}

using (StreamWriter sw = new StreamWriter("CDriveDirs.txt"))
{
    foreach (int[] xY in worlds)
    {
        sw.WriteLine(xY);
    }
}

using (StreamWriter sw = new StreamWriter("CDriveDirs.txt"))
{
    foreach (int[] xY in worlds)
    {
        sw.WriteLine(xY);
    }
}

using (StreamWriter sw = new StreamWriter("CDriveDirs.txt"))
{
    foreach (int[] xY in worlds)
    {
        sw.WriteLine(xY);
    }
}

?

J

Jimmacle•8/17/23, 1:30 AM

like

        f.Write(data[i][0]);
        f.Write(',');
        f.WriteLine(data[i][1]);

        f.Write(data[i][0]);
        f.Write(',');
        f.WriteLine(data[i][1]);

        f.Write(data[i][0]);
        f.Write(',');
        f.WriteLine(data[i][1]);

        f.Write(data[i][0]);
        f.Write(',');
        f.WriteLine(data[i][1]);

J

Jimmacle•8/17/23, 1:31 AM

obviously test to see which is actually faster, but this should avoid some string allocations

J

Jimmacle•8/17/23, 1:33 AM

using System.Diagnostics;

int[][] data = Enumerable.Range(0, 100000000).Select(_ => new[] { Random.Shared.Next(), Random.Shared.Next() }).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.CreateText("output.txt");
for (var i = 0; i < data.Length; i++)
{
    f.Write(data[i][0]);
    f.Write(',');
   f.WriteLine(data[i][1]);
}
Console.WriteLine(sw.Elapsed);

using System.Diagnostics;

int[][] data = Enumerable.Range(0, 100000000).Select(_ => new[] { Random.Shared.Next(), Random.Shared.Next() }).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.CreateText("output.txt");
for (var i = 0; i < data.Length; i++)
{
    f.Write(data[i][0]);
    f.Write(',');
   f.WriteLine(data[i][1]);
}
Console.WriteLine(sw.Elapsed);

using System.Diagnostics;

int[][] data = Enumerable.Range(0, 100000000).Select(_ => new[] { Random.Shared.Next(), Random.Shared.Next() }).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.CreateText("output.txt");
for (var i = 0; i < data.Length; i++)
{
    f.Write(data[i][0]);
    f.Write(',');
   f.WriteLine(data[i][1]);
}
Console.WriteLine(sw.Elapsed);

using System.Diagnostics;

int[][] data = Enumerable.Range(0, 100000000).Select(_ => new[] { Random.Shared.Next(), Random.Shared.Next() }).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.CreateText("output.txt");
for (var i = 0; i < data.Length; i++)
{
    f.Write(data[i][0]);
    f.Write(',');
   f.WriteLine(data[i][1]);
}
Console.WriteLine(sw.Elapsed);

gets me

start
00:00:10.2480180

start
00:00:10.2480180

J

Jimmacle•8/17/23, 1:35 AM

f.WriteLine($"{data[i][0]},{data[i][1]}");

f.WriteLine($"{data[i][0]},{data[i][1]}");

gets

00:00:12.6447007

00:00:12.6447007

J

Jimmacle•8/17/23, 1:35 AM

(these are very unscientific benchmarks)

A

Angius•8/17/23, 1:36 AM

Yeah, run it through Benchmark.NET to get info about allocations and all that jazz

J

Jimmacle•8/17/23, 1:37 AM

realistically disk write speed is gonna be more of a bottleneck than any micro-optimization here

W

WaffleDevsOP•8/17/23, 1:38 AM

Ah it worked in <1m

thank you!

W

WaffleDevsOP•8/17/23, 1:38 AM

W

WaffleDevsOP•8/17/23, 1:38 AM

Hahaha

W

WaffleDevsOP•8/17/23, 1:39 AM

Oh

J

Jimmacle•8/17/23, 1:39 AM

those must be small ints, my output file with random numbers is about 2GB

W

WaffleDevsOP•8/17/23, 1:39 AM

They are x/y coords. min is -8000 for both, max is +8000

J

Jimmacle•8/17/23, 1:40 AM

i wonder if parallelization would make it faster PepeHmmm

J

Jimmacle•8/17/23, 1:41 AM

my SSD should be able to do 2.5GB/s write

J

Jimmacle•8/17/23, 1:41 AM

can you even seek and write to the same file in parallel?

J

Jimmacle•8/17/23, 1:42 AM

i guess it wouldn't even apply here since the line length is variable

WWaffleDevs Oh

A

Angius•8/17/23, 1:42 AM

https://github.com/RolandPheasant/TailBlazer this is made for browsing logs, but also great for opening any text file that's too big for anything else. Can open tens of gigs of

.txt

.txt

files no problem

W

WaffleDevsOP•8/17/23, 1:43 AM

Ah ty, i was just using VSC and it crashed

JJimmacle can you even seek and write to the same file in parallel?

A

Angius•8/17/23, 1:43 AM

I don't think you can write to the same file from multiple threads

J

Jimmacle•8/17/23, 1:47 AM

2 seconds

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    unsafe
    {
        fixed (int* ptr = data[i])
        {
            f.Write(new ReadOnlySpan<byte>(ptr, 8));
        }
    }
}

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    unsafe
    {
        fixed (int* ptr = data[i])
        {
            f.Write(new ReadOnlySpan<byte>(ptr, 8));
        }
    }
}

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    unsafe
    {
        fixed (int* ptr = data[i])
        {
            f.Write(new ReadOnlySpan<byte>(ptr, 8));
        }
    }
}

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    unsafe
    {
        fixed (int* ptr = data[i])
        {
            f.Write(new ReadOnlySpan<byte>(ptr, 8));
        }
    }
}

J

Jimmacle•8/17/23, 1:48 AM

abandon text, embrace binary

J

Jimmacle•8/17/23, 1:48 AM

if you used an array of tuples you could just dump the whole block of memory into a file

J

Jimmacle•8/17/23, 1:49 AM

assuming you don't have any concerns about endianness

J

Jimmacle•8/17/23, 1:51 AM

which knocks it down to 0.5 seconds on my machine and pretty closely matches the theoretical max write speed of my SSD

J

Jimmacle•8/17/23, 1:52 AM

(int, int)[] data = Enumerable.Range(0, 100000000).Select(_ => (Random.Shared.Next(), Random.Shared.Next())).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.Create("output.bin");
unsafe
{
    fixed ((int, int)* ptr = data)
    {
        f.Write(new ReadOnlySpan<byte>(ptr, data.Length * 8));
    }
}
Console.WriteLine(sw.Elapsed);

(int, int)[] data = Enumerable.Range(0, 100000000).Select(_ => (Random.Shared.Next(), Random.Shared.Next())).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.Create("output.bin");
unsafe
{
    fixed ((int, int)* ptr = data)
    {
        f.Write(new ReadOnlySpan<byte>(ptr, data.Length * 8));
    }
}
Console.WriteLine(sw.Elapsed);

(int, int)[] data = Enumerable.Range(0, 100000000).Select(_ => (Random.Shared.Next(), Random.Shared.Next())).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.Create("output.bin");
unsafe
{
    fixed ((int, int)* ptr = data)
    {
        f.Write(new ReadOnlySpan<byte>(ptr, data.Length * 8));
    }
}
Console.WriteLine(sw.Elapsed);

(int, int)[] data = Enumerable.Range(0, 100000000).Select(_ => (Random.Shared.Next(), Random.Shared.Next())).ToArray();

Console.WriteLine("start");
var sw = Stopwatch.StartNew();
using var f = File.Create("output.bin");
unsafe
{
    fixed ((int, int)* ptr = data)
    {
        f.Write(new ReadOnlySpan<byte>(ptr, data.Length * 8));
    }
}
Console.WriteLine(sw.Elapsed);

J

Jimmacle•8/17/23, 1:55 AM

so the question is, is having it in text format (and a jagged array) worth it taking 5-20 times as long?

AAngius I don't think you can write to the same file from multiple threads

M

mtreit•8/17/23, 3:10 AM

Sure you can

M

mtreit•8/17/23, 3:11 AM

However you need to be very careful to not corrupt the hell out of the file

JJimmacle 2 seconds <:when:968512924698169415> ```cs using var f = File.Create("output.b...

A

Aaron•8/17/23, 6:44 AM

stop this immediately

J

Jimmacle•8/17/23, 6:44 AM

no

J

Jimmacle•8/17/23, 6:45 AM

gotta go fast

J

Jimmacle•8/17/23, 6:46 AM

actually i made a typo there didn't i

A

Aaron•8/17/23, 6:47 AM

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    f.Write(MemoryMarshal.AsBytes(data[i].AsSpan());
}

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    f.Write(MemoryMarshal.AsBytes(data[i].AsSpan());
}

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    f.Write(MemoryMarshal.AsBytes(data[i].AsSpan());
}

using var f = File.Create("output.bin");
for (var i = 0; i < data.Length; i++)
{
    f.Write(MemoryMarshal.AsBytes(data[i].AsSpan());
}

A

Aaron•8/17/23, 6:47 AM

at least make it safe

J

Jimmacle•8/17/23, 6:47 AM

i don't need no new fangled fancy memory apis

AAaron ```cs using var f = File.Create("output.bin"); for (var i = 0; i < data.Length; ...

M

mtreit•8/17/23, 7:48 AM

lol wtf is this

A

Aaron•8/17/23, 7:49 AM

the same thing jimmacle did in that first code block

❔ Whats the fastest way to write this data to a file? (C# Console App.)

Similar Threads

Similar Threads

Similar Threads