Peacocks revisited....
Before you allow yourself to get discoraged by the superficiality of it all...you might want to consider that far from superficial, what the ladies in your sphere of influence are responding to, perhaps goes a lot deeper than any of us ever stop to consider.
If you do a search on the courtship displays of males across a multitude of different animal species, you will find that the most wild and colorful representation of each species is found in the courting male, not the females, and not the babies. Going by the pictures of some of the animals I've seen, your discreet blue shirt with carefully rolled sleeves and buttoned collar is very tame by comparison. (it's a nice touch, BTW, aah....yes, I'm female :-)
It may be that we are every bit as programmed to offer and be receptive to displays of outerwear as our cousins in the animal kingdom. So next time you don your shirt to go for a stroll, be empowered by the thought that you are about to do the very same thing that (almost) all other males on the planet, human or otherwise, are doing at the same time, somewhere. See yourself ruffling your mane, revving up your testosterone and going out for the "kill", sort-of-speak. Needless to say, enjoy yourself!