Though oregano contains phtyoestrogens, there is no evidence that it increases estrogen levels in men or women. In fact, it might do just the opposite. Why would you want to avoid estrogen though? Men need a certain amount of estrogen for proper health function and lack of estrogen may increase the risk of fractures and osteoporosis. I personally use quite a bit oil of oregano and have yet to develop any man-boobs . . .
Perfectly safe? For all men? I cannot make that blanket of a statement, but as far as I know it is safe. Oregano is after all a natural herb and spice which has been used for many centuries for healing and cooking. By comparison, I don't think hardly any unnatural drugs are safe.