Pasteurized Milk can feed bad bacteria more than yeast or fungus. Lactose is more designed for bacteria than yeast. Looking at some microbiology experiments only 1 out of the 20 candida strains tested could ferment lactose efficiently. Depends on the dairy and whats added to it. Raw butter and raw milk is more designed for bacteria than yeast or fungus.
However, fungus can adapt overtime to utilize just about anything. Especially when there is very little bacteria to keep it in check(keep it in its yeast form). Given enough time I'm sure fungus will be able to utilize lactose more efficiently, but in the short term not as much--at least according to the experiment.
I've had hearing loss due to ear infections when I was younger. Seems like the cause was bad bacteria digesting the undigested lactose--due to pasteurization/ preservatives (enzyme inhibitors in food).