You should not worry what others think. I can imagine you are old enough to make your own sound decisions. Tell them you have a prior date; no one needs to know everything that goes on in your private life. I know sometimes it is hard to keep secrets from coworkers but realize they are not your friends and you do not have to tell them anything.