C#•3y ago

❔ Mapping common primitive types with protobuf & grpc

I have a bunch of generated classes from proto that contain semi-pritmitive data types, for example Vector3. In every project I have to define (copy-paste) mapping code from the generated proto Vector3 to the proto Vector3.

Are there any best practices doing this kind of stuff? I found auto mapping libraries but those still require me to define a mapping and I'd be repeating the mapping definitions in every project where I use the generated files.

JakenVeina•10/6/23, 8:33 PM

so, you're using a proto file as your source of truth?

AkroMentosOP•10/6/23, 8:43 PM

AkroMentosOP•10/6/23, 8:47 PM

I believe the System.Numerics.Vector3 is the best example I have.
imagine the following proto definition:

At some point in the code after receiving this proto message you'd want to tranform the proto Vector3 into a System.Numerics.Vector3 to use it for calculations / call libraries, etc. In every project that I use the above proto file I have to copy-paste a mapping function that given a Namespace.Of.Just.Generated.Proto.Vector3 into a System.Numerics.Vector3.

JakenVeina•10/6/23, 9:10 PM

what's generating these models?

AkroMentosOP•10/6/23, 9:14 PM

I use Grpc.Tools

JakenVeina•10/6/23, 9:21 PM

which generates from what?

AkroMentosOP•10/6/23, 9:22 PM

proto

AkroMentosOP•10/6/23, 9:22 PM

ah you meant source of truth for the code generation...

AkroMentosOP•10/6/23, 9:24 PM

I edited OP because for some reason my brain wrote that I have generated classes from grpc instead of "i have generated classes from proto"

JakenVeina•10/6/23, 9:26 PM

is Grpc.Tools a source generator? or just a scaffolding tool?

AkroMentosOP•10/6/23, 9:28 PM

it's generating code compile time so I believe it's using source generators, or something similar

AkroMentosOP•10/6/23, 9:28 PM

but no idea how they choose to implement codegen

JakenVeina•10/6/23, 9:29 PM

well, that's what I was getting at

JakenVeina•10/6/23, 9:30 PM

I dunno, unless they specifically have hooks for allowing you to define your own type mappings, you're probably out-of-luck

AkroMentosOP•10/6/23, 9:32 PM

Even if it did I wouldn't want to define any project/tool specific thing in the proto file as these are shared between completely different teams using different tech stack

JakenVeina•10/6/23, 9:37 PM

right, I'm talking about Grpc.Tools

AkroMentosOP•10/6/23, 9:40 PM

so what you're suggesting is that if there was a way it'd be best if instead of the generated Vector3, the code generator could already put System.Numerics.Vector3 in there.

just imagingin how that could possible work, at that point the generated code couldn't possibly call the proto write on that vector3 because obviously System.Numerics.Vector3 is not a proto class.

There'd need to exist a mapping still somewhere

JakenVeina•10/6/23, 9:43 PM

right

JakenVeina•10/6/23, 9:44 PM

it's entirely possible, but I have no idea if Grpc.Tools supports anything like that

JakenVeina•10/6/23, 9:44 PM

you'll have to read their docs

JakenVeina•10/6/23, 9:45 PM

or, more likely, as it looks like they don't really HAVE any docs, you'll have to dive through source code

AkroMentosOP•10/6/23, 9:45 PM

that's fine, can you please give me the tool whith which you're familiar where it's possible? I can take inspiration from that

AkroMentosOP•10/6/23, 9:46 PM

(and yes grpctools has basically no docs..)

JakenVeina•10/6/23, 9:46 PM

it's... not something you can take inspiration from

JakenVeina•10/6/23, 9:46 PM

Grpc.Tools is either extensible, or it isn't

AkroMentosOP•10/6/23, 9:48 PM

I mean.. it's open source I can fork / rewrite do anything, I can also switch over to what you're using

AkroMentosOP•10/6/23, 9:49 PM

if it has the featureset that I need... I'm not married to grpc.tools...

JakenVeina•10/6/23, 10:38 PM

what do you mean "what I'm using"?

JakenVeina•10/6/23, 10:38 PM

I'm not "using" anything

AkroMentosOP•10/7/23, 5:28 AM

sorry, I thought you had something specific in mind when you said

it's entirely possible, but I have no idea if Grpc.Tools supports anything like that

it came across that you are also using protobuf and know of a library or tool that supports what I need or have solved this issue otherwise

JakenVeina•10/7/23, 5:39 AM

I've used protobuf a bit before, but the other way around

JakenVeina•10/7/23, 5:39 AM

C# models as the source of truth, with the option to generate proto files for other languages, if I want

JakenVeina•10/7/23, 5:40 AM

and I'm vaguely familiar with a few source gen libraries

AkroMentosOP•10/7/23, 5:40 AM

hm, interesting approach, unfortunately I get the proto files so I have to use that as source

JakenVeina•10/7/23, 5:40 AM

right

JakenVeina•10/7/23, 5:40 AM

you could always write your own source gen wraffSmug

AkroMentosOP•10/7/23, 5:41 AM

I do that already for other stuff, and I would do it here as well. My main issue is that I have no idea how a solution could look like

AkroMentosOP•10/7/23, 5:42 AM

at the end of the day that mapping has to be defined somewhere

JakenVeina•10/7/23, 5:42 AM

right

JakenVeina•10/7/23, 5:42 AM

so, how does the source gen lib pick up on the proto files to gen for?

AkroMentosOP•10/7/23, 5:43 AM

AkroMentosOP•10/7/23, 5:44 AM

then it's either some msbuild step of source generators that do the magic

JakenVeina•10/7/23, 5:44 AM

sure

JakenVeina•10/7/23, 5:44 AM

so, there could be a config file that you include in a similar manner

JakenVeina•10/7/23, 5:44 AM

or just config that you specify directly in the MSBuild

JakenVeina•10/7/23, 5:44 AM

the config itself ought to be fairly simple, right?

JakenVeina•10/7/23, 5:44 AM

you ultimately just need to say "for type X in protobuf, translate it to System.Numerics.Vector3"

JakenVeina•10/7/23, 5:46 AM

and the source gen would need to do a little but of analysis to find that system type and make sure it's compatible and emit whatever translation code might be necessary

JakenVeina•10/7/23, 5:46 AM

or, like, emit some kind of class for each mapping

JakenVeina•10/7/23, 5:46 AM

whatever the appropriate code would be if you were writing it yourself