Set bonus combo breaker?

If another piece is too close to a matched set, does that ruin the look even (or especially) if it is very close? The hamilton brown used by carhartt wip is slightly darker and richer than the tobacco canvas used by taylor stitch, although the difference isn't nearly as pronounced under normal lighting. I didn't get the hat to wear with the set, but could I get away with it?