Absolutely right on!
Yup, confidence is the thing. One of the sexiest men I ever met was also quite a fatty, not great looking, a teddy bear of sorts, but very , very manly. He certainly wasn't an A-hole, but he was indeed very masculine. Well, he had my head spinning for quite a while until I realized that for a number of other reasons, he was not the one that I would pick for a partner. He was actually very hurt and I felt awful, but better to break in the beginning than get stuck into the wrong situation.
Anyway, the point here is that definitely, is is something that is being projected from the inside out that exudes a lack of self confidence, fear, or something. It just can't be a matter of looks or something supperficial, unless hygiene is lacking altogether.
Don''t you have a good friend with whom you could discuss this? Someone who knows you well might add some insight.