I just need to add my two cents here. I joined today because I am in the process of going raw diet.
Anyway, my husband is 59 and has been a vegeterian for 30 years. He happily eats tofu just about every night of his life. He is a creature of habit.
He also has the highest libido of any male I personally, have ever met. I used the hide username because it is a little embarrassing to me to post that as my first message. but I searched for something soy-related and ran into this thread.
He may be an exception to this rule too, as you are. Or this may not be valid Science in this case.