Agnostic client-side serialization of custom types

Hi, I've noticed a potential trend in G.V() of serialization issues for types that appear to be custom and therefore not deserializable from G.V()'s perspective. Where things get tricky is from G.V()'s perspective we need to somehow be able to connect to any database regardless of this type of customer specific situation. Is there a way to configure a fallback deserializing mechanism client-side (e.g. GraphBinary config or otherwise) that automatically deserializes a result as string (or some other default) in the absence of an appropriate serializer being available?
Solution:
Okay so final update on this, hopefully. I'm coming to the realisation that this is just not straight forward in any sense, even with some sort of fallback mechanism there would still be the issue of determining which/how many bytes should be read in the buffer for the incoming type, which i guess would be nearly impossible without actual knowledge of the structure expected.
Jump to solution
16 Replies
triggan
triggan•2y ago
I think this was really the intent of GraphSON, where the type information was included in the more verbose output.
gdotv
gdotv•2y ago
Yeah, ive had somewhat mixed results with graphson, im unsure if it's server related but it definitely tends to result in similar deserialisation issues My thinking is along the lines of implementing a version of say graphbinary within g.v that would either silently fail on deserialisation failure or handle the result in a different way somehow
spmallette
spmallette•2y ago
we purposely didn't do a toString() serialization of GraphBinary. i see how it could be useful for this case you describe but i think it's less useful more generally. maybe there could be a configuration for it, but if the server isn't configured with it that's where you will have a problem, right? or are you having a problem with like a JanusGraph RelationIdentifier coming back from the server and you can't do anything with it (i.e. ends in error)?
gdotv
gdotv•2y ago
yeah, i get that. my challenge is really just that a single serialization failure will throw the entire query out from G.V()'s perspective so im thinking it'd be worthwhile having a sort of soft fail mechanism where if deserialization fails a bespoke type gets returned that from G.V()'s perspective just signifies a failure to deserialize a specific value so in essence would i be able to implement a class inheriting graphbinary and overriding its fatal exception behaviour on failure to deserialize an unknown type
spmallette
spmallette•2y ago
wow, are you like actually asking for a change to TinkerPop right now? 🙂
gdotv
gdotv•2y ago
Now come to think of it that could just a change to the driver's implementation of graphbinary, for instance. I have to admit since gryo got deprecated gdotv already embeds these classes in so my mind turned to just making my own implementation within gdotv but if there was value outside of my weird stuff then sure!
spmallette
spmallette•2y ago
i dont think we want a toString() sort of default. i'm never sure what toString() contains and it might not be appropriate (could even contain a security risk). I suppose if the DataType.CUSTOM isn't found we could return an "UNKNOWN" that perhaps includes the name of the custom type and its BLOB.
gdotv
gdotv•2y ago
Yup, that's what i had in mind ill give it a quick shot on my end and let you know how that goes, ive already got these pesky datastax enterprise graph custom types in mind for this type of situation.
gdotv
gdotv•2y ago
im noticing the existence of a withFallbackResolver mechanism on type registry which sounds like it does more or less what im looking for, ill give it a shot (https://github.com/apache/tinkerpop/blob/master/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/binary/TypeSerializerRegistry.java#L227)
GitHub
tinkerpop/TypeSerializerRegistry.java at master · apache/tinkerpop
Apache TinkerPop - a graph computing framework. Contribute to apache/tinkerpop development by creating an account on GitHub.
gdotv
gdotv•2y ago
so just a little progress update here
gdotv
gdotv•2y ago
the fallbackResolver mechanism felt like it should work but im finding that it won't for all cases, specifically when handling classes flagged as "DataType.Custom". I guess the question I'm asking myself now is this: Should the getSerializerForCustomType method implement a similar fallback mechanism using the optional fallbackResolver as the getSerializerForType method? (https://github.com/apache/tinkerpop/blob/master/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/binary/TypeSerializerRegistry.java#L399 for reference)
GitHub
tinkerpop/TypeSerializerRegistry.java at master · apache/tinkerpop
Apache TinkerPop - a graph computing framework. Contribute to apache/tinkerpop development by creating an account on GitHub.
gdotv
gdotv•2y ago
due to the class structure at the moment I can't really extend it and override that specific method, so what im going to try in the meantime is create my own "FailSafeGraphBinarySerializerV1" that just implements a custom implementation of TypeSerializerRegistry, which ill just call "FailSafeTypeSerializerRegistry" and which will just be a shameless copy paste of the code with that extra 2 lines of code
Solution
gdotv
gdotv•2y ago
Okay so final update on this, hopefully. I'm coming to the realisation that this is just not straight forward in any sense, even with some sort of fallback mechanism there would still be the issue of determining which/how many bytes should be read in the buffer for the incoming type, which i guess would be nearly impossible without actual knowledge of the structure expected.
gdotv
gdotv•2y ago
Some interesting implementation of this is datastax's own graphbinary serializer for its custom user defined types (UdtValue as they're called) which I guess are the closest thing to a self assessed definition of the incoming byte data based on the defined data schema
spmallette
spmallette•2y ago
i will try to find some time to look into it this week
gdotv
gdotv•2y ago
definitely no rush, im going to keep investigating because this is quite fun and put findings here
Want results from more Discord servers?
Add your server