The only way to find out is to be DIRECT with him.
Next time he does that ask him, "why do you sound hesitant about it, i sense something in your voice that you seem weird about it, Im I right about this?" if he says no then you dont have a problem just keep livin your life. If he says yes then ask him whats the problem.