Yeah, we rarely eat tofu and we never drink soy milk. But we put soy sauce in everything lol
I think it really depends on the individual. Sensitivity/allergy to soy seems to run in my family. I tried eating natto (fermented soybeans) for the beneficial nattokinase but my thyroid turned into a goiter after only a few beans. And those beans were organic too.
My cousin is even more sensitive to soy; her throat closes up every time she accidentally eats tofu.