The WORST thing about drinking milk is the hormone injections that the cows get, obvously, especially if you're a man. But beyond that, you just have to make a judgement for yourself. I don't think it would be smart to consume an endless amount of milk and dairy products, as MOST people do. Far from it! However, I see no problem with adding some farm-raised cow's milk to a cup of tea every once in a while....
What I'm saying is it's silly to start removing every trace of dairy product from your diet all at once. It is better to treat dairy products as a treat, which is really what they are.