Brief hug, brief touch on the shoulder or arm, all that is fine, especially with someone I've known for some years & see on a regular, if infrequent, basis. It would mean we share a common interest & have developed a level of familiarity with each other whereby we know there's a basis for the (brief) hug. It's the lingering touch, or the kind of touch that makes you frown because it's not appropriate to the circumstance, or to the level of your relationship, that is problematic.

When a man behaves within accepted societal norms, there's nothing to remark on. When his behavior draws attention, such that people who are disinterested observers start commenting publicly on it—well, by definition that means he has been acting outside of accepted norms, no? Society eventually notices outré behavior and calls it out. Even when it's not as bad as other bad things.