getting off milk and dairy

the idea that milk is bad for you has been growing in the back of my mind, i have been reading alot of bits of articles here and there, and its all adding up.

i might not be ready to cut it all out completly but i am interested in going one week with absolutly no dairy. taking the milk out of my cereal in the morning is going to be a challenge, so is cooking without cheese. but for a week i can simply choose recipes that do not include cheese and it should be fine.

so is it mainly milk that is really bad, or is cheese up there? goat cheese, feta cheese, mozzarela, etc.

what about butter and yogourt, that is dairy, do i cut those out?

i was thinking of replacing my morning fiber cereal with some fruit/yogourt smoothies, but how do i replace the fiber. i need that 5-7g every morning for things to function as they should.