I hear that too much soy can contribute to cancer,osteoporosis(i think because it mimics estrogen), is bad for the brain, hard on the liver, and can impair thyroid function. I don't claim any of this to be true, these are just things that i've read.
Also, I would assume that people who experiance these problems due to soy intake are replacing everything with soy products.