Interesting. I'm a lot like you MsH. Touchy feely only with people I know and trust totally. (mostly family) some people do seem to "pull my plug" though and I have to be careful b/c even if I care a/b that person, too much time or conversation (even short ones) can be a drain! If I hug her I can feel a sort of sucking of my energy!! I've always been this sensitive though.
I read somewhere that people that are as empathetic and feeling and in-tune with other people's feelings are more susceptible to illness. Kinda like our defenses are more "open" so we are more prone to things. Senstive in emotional and also physical (and spiritual of course) ways can be a double-edged sword. It takes learning to "wield this sword"!
I'm not all there yet, but part of it is letting go of fear even when we are around those draining people. Of course set limits, but don't freak out if you are put in contact with them. I've noticed if I freak out that they may "steal" some of my energy, it gets worse and takes longer to recover myself. So now I'm just like, it's ok!! I can give a little graciously without fear, then the recover time is like nothing! :)