If you let others bother you then they win. It is best to know yourself and let them carry the burden of guilt and backbiting. You are bigger than that.
Being sensitve is a good thing, but be sensitive and feel sorry for these people who talk behind your back, they are petty.