Topics

C#•9mo ago

❔ Mapping common primitive types with protobuf & grpc

I have a bunch of generated classes from proto that contain semi-pritmitive data types, for example Vector3. In every project I have to define (copy-paste) mapping code from the generated proto Vector3 to the proto Vector3. Are there any best practices doing this kind of stuff? I found auto mapping libraries but those still require me to define a mapping and I'd be repeating the mapping definitions in every project where I use the generated files.

37 Replies

JakenVeina•9mo ago

so, you're using a proto file as your source of truth?

AkroMentos•9mo ago

no I believe the System.Numerics.Vector3 is the best example I have. imagine the following proto definition:

message Vector3 {
  float X = 1;
  float Y = 2;
  float Z = 3;
}
message EventWithPosition {
   Vector3 Position = 1;
   string something = 2;
   // ...
}

message Vector3 {
  float X = 1;
  float Y = 2;
  float Z = 3;
}
message EventWithPosition {
   Vector3 Position = 1;
   string something = 2;
   // ...
}

At some point in the code after receiving this proto message you'd want to tranform the proto Vector3 into a System.Numerics.Vector3 to use it for calculations / call libraries, etc. In every project that I use the above proto file I have to copy-paste a mapping function that given a Namespace.Of.Just.Generated.Proto.Vector3 into a System.Numerics.Vector3.

JakenVeina•9mo ago

what's generating these models?

AkroMentos•9mo ago

I use Grpc.Tools

JakenVeina•9mo ago

which generates from what?

AkroMentos•9mo ago

proto ah you meant source of truth for the code generation... I edited OP because for some reason my brain wrote that I have generated classes from grpc instead of "i have generated classes from proto"

JakenVeina•9mo ago

is Grpc.Tools a source generator? or just a scaffolding tool?

AkroMentos•9mo ago

it's generating code compile time so I believe it's using source generators, or something similar but no idea how they choose to implement codegen

JakenVeina•9mo ago

well, that's what I was getting at I dunno, unless they specifically have hooks for allowing you to define your own type mappings, you're probably out-of-luck

AkroMentos•9mo ago

Even if it did I wouldn't want to define any project/tool specific thing in the proto file as these are shared between completely different teams using different tech stack

JakenVeina•9mo ago

right, I'm talking about Grpc.Tools

AkroMentos•9mo ago

so what you're suggesting is that if there was a way it'd be best if instead of the generated Vector3, the code generator could already put System.Numerics.Vector3 in there. just imagingin how that could possible work, at that point the generated code couldn't possibly call the proto write on that vector3 because obviously System.Numerics.Vector3 is not a proto class. There'd need to exist a mapping still somewhere 😐

JakenVeina•9mo ago

right it's entirely possible, but I have no idea if Grpc.Tools supports anything like that you'll have to read their docs or, more likely, as it looks like they don't really HAVE any docs, you'll have to dive through source code

AkroMentos•9mo ago

that's fine, can you please give me the tool whith which you're familiar where it's possible? I can take inspiration from that (and yes grpctools has basically no docs..)

JakenVeina•9mo ago

it's... not something you can take inspiration from Grpc.Tools is either extensible, or it isn't

AkroMentos•9mo ago

I mean.. it's open source I can fork / rewrite do anything, I can also switch over to what you're using if it has the featureset that I need... I'm not married to grpc.tools...

JakenVeina•9mo ago

what do you mean "what I'm using"? I'm not "using" anything

AkroMentos•9mo ago

sorry, I thought you had something specific in mind when you said

it's entirely possible, but I have no idea if Grpc.Tools supports anything like that

it came across that you are also using protobuf and know of a library or tool that supports what I need or have solved this issue otherwise

JakenVeina•9mo ago

I've used protobuf a bit before, but the other way around C# models as the source of truth, with the option to generate proto files for other languages, if I want and I'm vaguely familiar with a few source gen libraries

AkroMentos•9mo ago

hm, interesting approach, unfortunately I get the proto files so I have to use that as source

JakenVeina•9mo ago

right you could always write your own source gen wraffSmug

wraffSmug

AkroMentos•9mo ago

I do that already for other stuff, and I would do it here as well. My main issue is that I have no idea how a solution could look like at the end of the day that mapping has to be defined somewhere

JakenVeina•9mo ago

right so, how does the source gen lib pick up on the proto files to gen for?

AkroMentos•9mo ago

<Protobuf Include="../../proto/common/*.proto" ProtoRoot="../../proto" />

<Protobuf Include="../../proto/common/*.proto" ProtoRoot="../../proto" />

then it's either some msbuild step of source generators that do the magic

JakenVeina•9mo ago

sure so, there could be a config file that you include in a similar manner or just config that you specify directly in the MSBuild the config itself ought to be fairly simple, right? you ultimately just need to say "for type X in protobuf, translate it to System.Numerics.Vector3" and the source gen would need to do a little but of analysis to find that system type and make sure it's compatible and emit whatever translation code might be necessary or, like, emit some kind of TypeConverter<T> class for each mapping whatever the appropriate code would be if you were writing it yourself

AkroMentos•9mo ago

ooo that's clever, like generate the converters as well

JakenVeina•9mo ago

if that's what it would take

AkroMentos•9mo ago

protobuf generates partial classes so I can also generate implicit conversions "into" the proto classes

JakenVeina•9mo ago

like proxy properties? well, that sounds like a reasonable-enough solution maybe not ideal

AkroMentos•9mo ago

that or just simply

 public static implicit operator Generated.Protobuf.Vector3(System.Numerics.Vector3 val)

 public static implicit operator Generated.Protobuf.Vector3(System.Numerics.Vector3 val)

so you'd still see the generated type in the protobuf but that can be implicitly converted to the desired domain type

JakenVeina•9mo ago

public class GeneratedMessageType
{
    public GeneratedVectorType Vector { get; set; }

    public Vector3 PrimitiveVector
    {
        get => Vector.Convert();
        set => Vector = value.ConvertBack();
    }
}

public class GeneratedMessageType
{
    public GeneratedVectorType Vector { get; set; }

    public Vector3 PrimitiveVector
    {
        get => Vector.Convert();
        set => Vector = value.ConvertBack();
    }
}

yeah I'd probably be a little more inclined to use an explicit property than an implicit cast those conversions can probably get expensive pretty quickly, so I'd rather have them be very visible

AkroMentos•9mo ago

good point, but will have to look out for name collisions

JakenVeina•9mo ago

it still seems like a common-enough scenario that Grpc.Tools ought to be extensible for you to put your own mappings in probably worth asking on their github page

AkroMentos•9mo ago

it doesn't have to be, I have a code generator for other purposes where I already do my own .proto -> .desc -> c# code, I can just hook into that and generate these mappings from a config file as well

JakenVeina•9mo ago

and if they don't support it, you could always contribute it yourself

AkroMentos•9mo ago

that way I'm still not married to grpc.tools

Accord•9mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

We are a programming server aimed at coders discussing everything related to C# (CSharp) and .NET.

52KMembers

View on Discord

Want results from more Discord servers?

Add your server

More Posts

Trying to use .Net 7 SDK in VS2022 17.8 Preview 2Hello! I'm incredibly confused, I'm trying to build a .Net7 sdk app in vs2022 17.8 preview 2 and it ❔ Which C# nethost file to include in c++ host applicationI am fallowing [this tutorial](https://learn.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosti ❔ Disable specific formatting rule only in one class in RiderSo, I have a class that is a bit pathological in that it has a bunch of one-liner declarations. Ride need helpd ❔ `required` keywordwhat is the purpose of the `required` keyword when we can enforce a property to be set through the ✅ C# best practices clarificationBy convention, which is preferable to use? Getter and Setter methods or properties? Are the equivale ❔ Returning a bool value when searching for the status of a service running on a remote serverHowdy, I am trying to return the status of a remote service. I can get it to work fine with somethi ✅ What does this property do?The backing field is `readonly`❔ Enum with dataI'm working on a lexer, and I'm encountering this problem where I need to associate different data w ❔ A Dictionary<Type, List<T>> that bakes the list for the type hierarchy (for fast Get calls)Does C# have an object that does this? When you try to Get/TryGet from the map, it will check if the