I totally empathize with what you describe. I've heard it described that people's energies (thoughts, emotions, whatever) can sort of jump between each other and so one can be affected by it.
I've heard Black Tourmaline can provide protection, and it seems to help for me. There are also Agua Sulis Stones (have you heard of them?). They are a type of crystal with which people have had profound experiences. It seems to help with getting re-balanced.