C#•3y ago

❔ Read large data over TCP

Team need inputs on reading large data using TCPClient.
I need to read data using TCPClient , I am sending a command and receiving data. but not sure whether I have read complete data. How do i check whether all the data has been read and how do I get the buffer size at receiver end ..can I use int.Max like Byte[Int32.Max]. But i noticed I am not getting the whole data at once.Should I loop it until I get the complete data?

using (TcpClient client = new TcpClient("192.168.100.123", 10001))
{
using (NetworkStream stream = client.GetStream())
{
using (StreamReader reader = new StreamReader(stream))
{

byte[] outStream = System.Text.Encoding.ASCII.GetBytes(";");
stream.Write(outStream, 0, outStream.Length);

// Buffer to store the response bytes.
Byte[] data = new Byte[5000]; // how do i know this size

//string content = new StreamReader(stream, Encoding.Unicode).ReadToEnd();

Int32 bytes = stream.Read(data, 0, data.Length);
var responseData = System.Text.Encoding.ASCII.GetString(data, 0, bytes);
}
}
}

deprecatio•9/8/23, 7:16 AM

format the code using triple backtick

SinFluxx•9/8/23, 7:16 AM

$code

SSinFluxx $code

MODiX•9/8/23, 7:16 AM

To post C# code type the following:
```cs
// code here
```

Get an example by typing

$codegif

$codegif

in chat

If your code is too long, post it to: https://paste.mod.gg/

deprecatio•9/8/23, 7:17 AM

can I use int.Max like Byte[Int32.Max]

please don't

deprecatio•9/8/23, 7:19 AM

How do i check whether all the data has been read

tcp sends packets
so yes you would have to continue to read all the packets
it depends on how your connection works, if it's one connection per request or if it's permanent
in the last case, you could send the size of the object before sending the object
so that the receiving part knows how much to read

Pobiega•9/8/23, 7:28 AM

Is it just straight raw TCP data, or is some kind of messaging protocol being used, like MLLP?

SShiv Team need inputs on reading large data using TCPClient. I need to read data usi...

TeBeCo•9/8/23, 8:53 AM

you're askign how to write a Protocol over TCP

TeBeCo•9/8/23, 8:54 AM

with RAW TCP there's no way to answer the question

TeBeCo•9/8/23, 8:54 AM

it's impossible

TeBeCo•9/8/23, 8:54 AM

you have to write your own "layer" for how to delimet "you type of packet" A from "you type of packet" B

TeBeCo•9/8/23, 8:54 AM

A and B beign 2 packet your app created (not TCP packet)

TeBeCo•9/8/23, 8:55 AM

creating that kind of packet is up to you to be able to delimit them

TeBeCo•9/8/23, 8:55 AM

eg:

is thatraw binary data ... like compressed data using MessagePack or protobuff
if that Text (like JSON)
etc ....

TeBeCo•9/8/23, 8:56 AM

you generally don't use the same approach for that "protocol writer/reader"

TeBeCo•9/8/23, 8:56 AM

for binary you generally can prefix the data by an

int

int

which contain the SIZE of your packet

TeBeCo•9/8/23, 8:57 AM

meaning you

first read an
```
int
```
```
int
```
=>
```
messageLength
```
```
messageLength
```
then read until you have
```
messageLength
```
```
messageLength
```
byte read

TeBeCo•9/8/23, 8:57 AM

there's literally an existing primitive for that in .net

TeBeCo•9/8/23, 8:57 AM

it's named

System.IO.Pipe

System.IO.Pipe

TeBeCo•9/8/23, 8:57 AM

$pipe

TTeBeCo $pipe

MODiX•9/8/23, 8:57 AM

see $pipeline

TeBeCo•9/8/23, 8:57 AM

$pipeline

TTeBeCo $pipeline

MODiX•9/8/23, 8:57 AM

(Contains example for Socket

TCP Echo

TCP Echo

)
https://devblogs.microsoft.com/dotnet/system-io-pipelines-high-performance-io-in-net/
https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines

if you're handling data from/to a stream, and dealing with intermediate buffer / parser / writer, you probably should consider using

System.IO.Pipelines

System.IO.Pipelines

It's handling internally

temporary buffer
memory pool to re-use memory and avoid allocation
accumulate buffer until you decide you have enough data to do something (deserialize ?)

There's dedicated extensions for

Stream

Stream

for specific use case like :
(Contains a sample to read file / dump to console)
Reading (eg: to a file):

using var stream = File.OpenRead("lorem-ipsum.txt");
var reader = PipeReader.Create(stream);

using var stream = File.OpenRead("lorem-ipsum.txt");
var reader = PipeReader.Create(stream);

Writing (eg: to Console):

var writer = PipeWriter.Create(Console.OpenStandardOutput(), new StreamPipeWriterOptions(leaveOpen: true));

var writer = PipeWriter.Create(Console.OpenStandardOutput(), new StreamPipeWriterOptions(leaveOpen: true));

ShivOP•9/8/23, 12:37 PM

@TeBeClone @Pobiega Sorry for late response. I am not sure about MLLP. But we have a Command to Query Real Time Sensor Values i.e, "!" and ";" for download of data

ShivOP•9/8/23, 12:37 PM

Real time data is fine, I am getting 35 char of data which I am processing

ShivOP•9/8/23, 12:41 PM

but for download I am not sure how much is the accumulated data... Is there we can check before .
Can I use something like this :

 int responseBytes = 0;
                            do
                            {
                                responseBytes = stream.Read(data, 0, data.Length); // store the response to the buffer
                                responseData += System.Text.Encoding.ASCII.GetString(data, 0, responseBytes);

                            } while (responseBytes > 0);

 int responseBytes = 0;
                            do
                            {
                                responseBytes = stream.Read(data, 0, data.Length); // store the response to the buffer
                                responseData += System.Text.Encoding.ASCII.GetString(data, 0, responseBytes);

                            } while (responseBytes > 0);

Ddeprecatio > can I use int.Max like Byte[Int32.Max] **please don't**

ShivOP•9/8/23, 12:53 PM

I have no control over sending part...Its a sensor data which I am reading... so only receiving part can i read until stream.Read is 0 ?

Pobiega•9/8/23, 1:12 PM

in general, what you would do is read until either your buffer is full or you've found an "end of packet" signifier

Pobiega•9/8/23, 1:12 PM

if the buffer is full, you're in trouble - it means a single packet was larger than your buffer. There are ways to get around this, like dumping the buffer somewhere else, flushing it and keep reading, but it gets annoying

Pobiega•9/8/23, 1:13 PM

so what you'd usually do is just set a somewhat large buffer, ideally large enough to store the largest possible packet

Pobiega•9/8/23, 1:13 PM

but if thats an unknown size, you'll have to guesstimate

ShivOP•9/8/23, 1:26 PM

OK Ending part is clear now.... I am stopping it when I get ";" but Will the Sender send the data randomly or is it in synchronous way.. because everytime I am getting different starting point of data...

ShivOP•9/8/23, 1:26 PM

will it get cleared at sender once we read the data? is it like that or is it device specific?

Pobiega•9/8/23, 2:04 PM

Your questions don't really make sense to me I'm afraid

Pobiega•9/8/23, 2:04 PM

think of the TCP stream as... well a stream

Pobiega•9/8/23, 2:04 PM

it sends one byte after the other

Pobiega•9/8/23, 2:04 PM

the trick is knowing when it stops

Pobiega•9/8/23, 2:05 PM

so if my buffer is empty, and I see...

123456;789

123456;789

Pobiega•9/8/23, 2:05 PM

I know that 123456 is one message, and I have a partial(unfinished) message that starts with 789

Pobiega•9/8/23, 2:06 PM

what I would do while reading this is as soon as I see

I stop reading, build a finished package, clear my buffer and keep going

Pobiega•9/8/23, 2:06 PM

repeat forever

Pobiega•9/8/23, 2:07 PM

the sender will know how TCP works for sure, so it wont send random bytes

Pobiega•9/8/23, 2:07 PM

it will start a package, send it, then send the ending separator

PPobiega in general, what you would do is read until either your buffer is full or you've...

TeBeCo•9/8/23, 3:32 PM

in general, what you would do is read until either your buffer is full or you've found an "end of packet" signifier

you can't do that with binary protocol, it works if you're going text protocol

TeBeCo•9/8/23, 3:33 PM

and IIRC there's a general known weird

0x??

0x??

for that

TeBeCo•9/8/23, 3:34 PM

goal is to avoid conflict the moment you evolve your protocol and add

string

string

with contains a

TeBeCo•9/8/23, 3:34 PM

and LOL

TeBeCo•9/8/23, 3:37 PM

    // This record separator is supposed to be used only for JSON payloads where 0x1e character
    // will not occur (is not a valid character) and therefore it is safe to not escape it
    public const byte RecordSeparator = 0x1e;

    // This record separator is supposed to be used only for JSON payloads where 0x1e character
    // will not occur (is not a valid character) and therefore it is safe to not escape it
    public const byte RecordSeparator = 0x1e;

TeBeCo•9/8/23, 3:37 PM

SShiv OK Ending part is clear now.... I am stopping it when I get ";" but Will the Se...

TeBeCo•9/8/23, 3:38 PM

if you strictly rely on TCP, then packet order it mandatory

TeBeCo•9/8/23, 3:38 PM

if you're multiplexing yourself the data in the same tcp stream you need to deal with that yourself => add yourself the packet id + the index ID of that check un that packet and the index of that packet over all packet

❔ Read large data over TCP

Similar Threads

Similar Threads

Similar Threads