Round trip reflective serialisation with exact type matching
I need to serialise lists of heterogeneous
Objects. These lists represent arbitrary "lookups" into data. For example [John, 2000-01-01] when paired with a key like [name, dob] represents people named John born on 2000-01-01. These key and value lists are saved in a database and associated with other data.
The difficulty is that at runtime I only have a List<Object> for the values. I can save this as JSON in the database but when pulling it out of the database number types will not map to the original type. JSON does not differentiate numerical values so eveything becomes double. I need the round trip to be identical so the objects generated by the application and those that are saved can be compared for equality.
I'd be grateful for any possible solution. So far I've considered Message Pack, Cbor and custom encoders/decoders. I only need to handle most primitive types, dates, timestamps and a small selection of others.
A cleverer way to key rows of arbitrary data according to a subset of key value pairs would also work.
I think a custom format maybe be the best option here but I'm open to any suggestions.13 Replies
⌛ This post has been reserved for your question.
Hey @undoublethink! Please useTIP: Narrow down your issue to simple and precise questions to maximize the chance that others will reply in here./closeor theClose Postbutton above when your problem is solved. Please remember to follow the help guidelines. This post will be automatically marked as dormant after 300 minutes of inactivity.
JSON doesn't work with double, maybe your JSON deserializer does?
Using Object is weird and I think not recommended in Java for things like this.
Maps in Java have key value pairs
You can create a record:
And use that instead of Object
But it seems you lack some of the Java basics...
Maybe try an easier project first instead?
I appreciate your answer but I've been a professional Java software engineer for many years. My problem cannot be solved by records. By nature of the problem the data is untyped and unstructured. The specific example from my original question was merely a singular example.
How does your mapping of a field to its datatype work?
Sorry if I misjudged you, so the problem is that you lose type data when serializing?
Sorry, I don't follow exactly. There is no explicit mapping at the moment. I have it using JSON (via moshi) at the moment, which presumably uses reflection to get the type for serialisation.
The "data" that is destined to match these lookups comes from a database. Where the types are mapped according to the database schema (but then end up being "lost") when they're thrown into a general purpose "DataFrame" type. The values might be transformed however and can change type during processing
when serializing you can add the fully qualified classname as an element and use that to know what class to instantiate
You mean serialise with type information? Then yes, that's a possibility. Not ideal but a potential solution. I wouldn't use the class name though 🙂
Where does Java come into the picture?
Java processes the
DataFrame representation of rows. It's a data transformation and cleaning application. The goal is to apply audited, repeatable patches to data depending on the values of certain (arbitrary) columns.Damn that's tough
With JSON you can know what valuetype something is. But it hasn't a lot of valuetypes. For your niche type of application you would probably have to create your own format. But I'm gonna stop trying to help you now, since I don't think I have anything useful to say that could help you sorry.
That's okay, I appreciate it 🙂
💤 Post marked as dormant
This post has been inactive for over 300 minutes, thus, it has been archived.
If your question was not answered yet, feel free to re-open this post or create a new one.
In case your post is not getting any attention, you can try to use /help ping.
Warning: abusing this will result in moderative actions taken against you.