First of all, I feel for you. Many of us have been through these kind of situations. Most people cannot stand good, hard working, honest, clean people. They want you to be like them. I would say that if you have nothing to hide and if you have not done anything wrong, hold your head up and chest high and behave like they are all petty little beings that are just wasting their energy. You should not waste yours on them. However, I would find out what is the reason they are really doing this. They want something, one reason could be they want you to leave the complex, but then why. What could be their gain? I would not talk to anyone about their accusations. I think you do need friends at work so they do not get you fired by false stories. Do not get worked up or they will get you in a corner where they want you.
Also if you are comfortable try to be friends with these people (it has to be a front) and beat to their game. Poison them with their own medicine. For example if there are two people who are against you but are not so close to each other, find out dirt about those two and tell each of them what the other is saying wrong about each other. They will not like it and eventually they will quit. But you must be two steps ahead of them and comfortable in doing so. Find out where they work and find some dirt on them and spread some ugly stuff about them at their work place. See how they like it. Make sure they do not know who is spreading rumors about them. They will hate it.
Fight back, dear, I know these kind of meanies.