Just say NO. Trust your instincts, walk away and tell someone. That's what parents and teachers tell 5 yr old kids. Don't wait to make sure that someone is violating you. As soon as you feel uncomfortable its time to make that exit. If you feel that's what happening it most likely is.
You're so blessed to have a kind understanding wife. I hope you get therapy to help you find out why you're not able to walk away and save yourself from these situations.
If you have kids you would want them to be strong and stand up for themselves right?