✅ Windows Development NamedPipeServerStream in dll halting communications with client
Hi All,
I have a C# dll that I am making to side load with a game in roder to expotr data from the game to be processed by a python machine learning algorithm. Right now I am trying to use a NamedPipeServerStream but I am having unexpected issues with communications suddenly halting with no explanation as to why.
Above are my Read and Write functions (I am avoiding using StreamReaders and StreamWriters) that run in seperate threads to preserve performance at the core of the main dll. My python client is in my first reply.
When the program starts I get two data transmissions that are picked up by the client and one full read by the C# dll but then reading and writing stops on the C# side and as such the python also stops. I need serious help here as I have been bashing my head against a wall for days trying to figure it out. My assumptions is maybe the threads are being killed by some lurking AV process maybe?
131 Replies
Here is the python client
Here is the full file if anyone wants to see
what has debugging told you?
(also, do yourself a favor, and ditch the cutesy lock names, you're just setting yourself up for confusion later)
Hey man, don't diss gary
Nah but I will
From my understanding at this point, it writes successfully twice, and then writing just stops
Which makes me think that threads are being killed
That's what I see in the log
will debug it?
will change the names of the locks
When you say debug do you mean add breakpoints?
because I can't do that in my current config
yes
it get's sideloaded with a game and the game crashes if I try to apply break points
ah
Yeah, makes it tough, which is why im logging everything
And it is also a dll, I can't really run it independently
one thing i would try is removing the locks
rather than using a static that you're locking around, instead use a BlockingCollection
BlockingCollection let me search that up
It seems useful, my only thing is that I don't want to queue data, I just want to transmit the current data
It looks like a BlockingCollection is list like
I see so much wrong here but don't have the confidence to give any concrete advice. The fact you're running writing and reading on the same server on two separate threads in endless loops makes me doubt the data is being transferred correctly at all
Why would that be?
It's an InOut stream
The way you're serializing the data is weird, the fact you're not using streamwriter/reader is weird, the lock seems completely misplaced (someone correct me)
I am open to as much advice as you can give, I never learnt C# formally
The Write and Read functions are in a seperate thread, and the main thread runs PlayerUpdate and accesses those resources
so I need to lock them, otherwise the dll crashes instantly
But where you put the lock is completely bogus
To me
I lock the data when I access it?
I thought that was right
You're accessing locals only
In player update, the main thread
they're locking around read/write of
data
and input
, both of which are staticAsa for the stream reader and writer, they were causing issues
Hm. Would be more clear if a proper naming convention was followed, but unrelated I suppose
I have gotten successful results encoding them myself. It just only works twice before dying
YEs yes i know im sorry
I havent really done a lot of windows development and game based dev so this is a bit foreign to me
Like as in I am not a veteran
I can switch back to a StreamReader and StreamWriter
but that still begs the question as to why it's suddenly halting
I am wondering if it would be better to use a TCP/UDP socket
but it is not as fast as a pipe
I would really need to have all of this in front of me to test for myself. I just see a major issue in writing this data so often while the python side has to do much more work (sending as well), which to me means it simply cannot keep up
The way I would implement this is by making the client ask for data (send request, wait for response with data), but I'm sure there's ways without that
Maybe look into
BeginRead
?I tried to implement a flag based system
Have it be, data is sent, then we wait for a response
once we read a resnponse then we send a new packet
You should probably move sending data to an actual Unity method that runs within the frame rate of the game. So something like their
Update
method in a MonoBrhaviour
Instead of just on a random threadI did not code the game
You're modding it
Yes
You can do that
I am coding a dll in Visual Studio
The game doesn't have an official mod support
hence we access obfuscated objects
But that is not the issue, I know the data is retrieved successfully
Actually I wouldn't even need to do it that way
The issue is more that you're not getting accurate data. You might skip a frame of the game, or might send the data from the same frame twice
What I might do is implement read and write in a single thread
as Read is blocking
so it won't write until it has received a response
If you expect to always read once and then immediately write once, then you should just do that, yes
I expected you wanted to individually listen and individually write whenever you want
That is what I wanted
But the python side really doesn't suggest that
but I think this will work too
If I could thread python then I would
but then I have to implement multiprocessing
you can thread python
Yes, but it doesn't ACTUALLY run on seperate threads
the GIL doesn't apply with waiting for I/O so it won't block the thread
Wait actually?
Yep, the underlying CPython call to wait for the data will release the GIL allowing the other thread to run whatever it wants
:|
There's also asyncio to use await
The pipe server on the C# side is not async, but I think that shouldnt affect the python client
Man this shit hard
multi threading is hard but I'm unsure from a glance why you actually need it
I need it on the C# side because the main PlayerUpdate thread can't be blocking and absolutely cannot take very long
Otherwise it physically slows down the game
Which is why I thread Write and Read in C#
Difficult to believe that writing to a stream is that slow
I had it in the main thread and it brought the game down from 300 to 60 fps
Using a .NET task will also allow you to avoid a thread, at least an explicit one, it'll rely on the threadpool to complete
Not sure what this is sorry
Pipes and StreamReader/Writer can be asynchronous in C# allowing you use async/Tasks on the C# side. Use a new enough .NET version and it'll also support cancellation tokens
I am using 4.7.2
There's your first problem 🙂
Cant update
Old game
I would if I could
I believe you can still use async here with net472
I could be wrong but IIRC named pipes don't like you reading and writing on separate threads on the same pipe. It's been a while since I tested it but I recall having some troubles with it
I had issues with async but I can try it again
Hm, I will try implement them on the same thread
...or use two pipes
Can I do that?
That's what we've been saying
yep
use as many as you want
I think I misunderstood
A stream doesn't have an output lane and an input lane. Your write thread is clogging up the stream buffer, and your read thread is trying to consume it immediately (someone correct me?)
There's all sorts of conflicts going on
A named pipe can be bidirectional but I still recall having troubles trying to read and write on the same pipe end at the same time
I seeeee
I thought if I marked it bidirectional it would manage two seperate lanes
Ok I am trying now with two seperate pipes
If you are writing data and expect a response only after writing it, then do the read in the same action
If you have read/writes independent from each other then I would recommend two different pipes
Also we never saw the logs btw
Just, you know
Would have been good
I am trying with two rq, lemme see if I can make it work
Looking at the docs, .NET 4.7.2 supports ReadLineAsync, WriteLineAsync and a pipe created Asynchronously. The only thing you miss out is cancellation support but you don't have that anyway with manual threading
Ok this shit wont even get past the initialisation now
this is how I connect
It can't connect to the servers
So what do the logs show it is getting to? Is C# seeing the client's connecting, is Python not connecting to any pipe?
It is not connecting to any pipe
oh my god
hang on
hang on hang on
I just had to do this
instead of connection after making them both
This might be working
If this works I can finally stop coding C# and just focus on the reinforcement learning in python
I would recommend also using the
Byte
transmission mode and just making sure you use StreamReader
and StreamWriter
to read and write lines instead
I can't say I've ever had the need of using Message
mode as it has some funky semantics when it comes to the underlying read/writesOH MY GOD IT WORKS
THANK GOD
I did do this but it was causing issues
And use like json to serialize the data instead of some custom format
It should work just fine
All it does is add a line break to the end of your input
I will try switch over now that I have got it working
I just decode a string
I didnt htink that was that bad
with a comma delimiter
Just make sure to also readline on the python side. It might assume
\n
though. StreamWriter likely writes \r\n
I am using readline use
.strip() in python gets rid of \r\n in any case
I will try switch to a StreamWriter and StreamReader
excuse the pwsh here but it shows the .NET types and calls make here

I do see
using json is a nice next step as you don't need to worry about escaping the delimiter values, the serialization can easily convert to a structured value dealing with the escaping all for you on both sides
Lemme get a StreamWriter and Reader going first
then I will do json
ok StreamWriter and Reader working
what are you guys suggesting with json?
@jborean
for python a simple
json.loads(read_line)
how do I encode as json on this side tho
for C# there's
System.Text.Json
that should have a compatible package for net472 or netstandard2.0, otherwise newtonsoft.jsonUnity should have built-in JSON stuff
I am in Visutal Studio
Doesn't matter?
You're writing a mod
You have access to all the unity stuff you need
I don't have Unity stuff installed
I do not believe this game was made with unity
The game comes with it
Oh, I thought it was
I dont believe so
It is SpeedRunners
Oh it's xna
yeah
I guess stj then
So how are you suggesting I format it? Like this?
just as a dumb example
then on the python end I use json.loads(the_data_that_was_read_in)
structure it however you wish
Maybe not an anonymous object...
I'd definitely make this a struct type
A record even
so this first
then create a new GameData object
A struct specifically, not a class
But for this you probably need to include PolySharp
And bump the langversion
hang on back up a sec
1. Why does it need to be a "record"
2. What is PolySharp
3. What do you mean by bump the langversion
It doesn't, records are just shorter lol
ah gotcha
The alternative is
Does record exist in 4.7.2
I say don't bother, the gain isn't worth it
In your case you can probably just go with a mutable type (leave the
set
and just assign them in the object initializer)also System.Text.Json doesn't exist :|
Basically just replace
class
with struct
here and call it a day
You need to include it as a package
I had using System.Text.Json
That's not including a package
I see
I did it
hooray
ok now my only question is what this will look like on the python side
.
No but as in, the data itself
it is not a json format
now*
@ero
is this correct in terms of getting the data from the C# json transmission
Does it work?
I just wanted to ask before testing it because there is a bug with the API at the moment that if something crashes I have to reaquire some files
At this point it's just QOL
I'm not the right person to ask, I don't know python
Thank you all for all your help
You have been great
Yep it just becomes a dict and you lookup keys with
value["key"]
gotcha
And I assume similarly the other way around, I just create stuff in python and use json.dumps?
honestly
yep
For such a small usecase is it even worth using json
you can dump it straight to the pipe as well to avoid the extra allocation but this is getting into micro optimisation territory here
pipe.write(json.dumps(cmd).encode())
?
@jborean
it will convert the value of
result
and write it to the fd represented by pipe
ah i see
I still call pipe.flush() tho yes?
most likely
you probably also want a newline as well, I can't remember if
json.dump
has an optino to expose that
doesn't look like it so you'll have to do it after
Cool
and on the C# is there like a deserialisation?
yep
JsonSerializer.Deserialize<T>(value)
, you provide the class as T
and it'll do it for youT is the struct to deserliase to
and value is a string
yep
how fast is the deserialiser
Shit man I am getting compatibility errors
Which version of System.Text.Json should I use for .NET 4.7.2
I dropped json
I will figure it out in future
it's simple to migrate to
but it causes like 15 errors due to version differences