ISerializationSurrogate and collections issue

I'm trying to port old code that was using BinaryFormatter and I'm using ISerializationSurrogate to be able to deserialize old files and it works except for one thing - all collections (Dictionary, HashSet) are returned empty from SerializationEntry.Value. I know they are not empty because in old project when I deserialize directly into type it works fine.