I have been in a relationship too, married for 4 years...we had sex very less times..maybe 15 times in 4 years...For me, I decided i was settling for second best and walked out...marriage is sacred but so are my needs and my desire to have a relationship that is intimate on every level.
I think u need to figure out the answer for yourself...dont listen to too many people, go with what your heart tells you and ask for guidance if you believe in a higher power...