He sounds like a narcissist to me. It is never his fault, he has no scruples or morals. He sounds like he's really bad news. I suspect that the woman on the different shift may have left the job because of this guy. Do you really want to be the other woman? And...if he's cheating on his wife, you can guarantee at some point, he'll cheat on you.
Everything you've posted is a red flag telling you to run in the opposite direction of this guy. He is also technically sexually harassing you at work. If you want it to stop, tell him you are going to file a complaint with HR if he doesn't stop and leave you alone. If he doesn't FILE A COMPLAINT. Keep documentation of his harassment, noting dates and specific facts -- no emotions just he said this and he did that with the date.