The grains should be pretty stable and resistant to being ruined. Have you been using these grains for awhile? How big a batch did you try to make? Maybe it was too big and slightly contaminated to start? I'd rinse the grains real well in milk (I stir them gently in a bowl, then strain them) a few times and then try again with a much smaller quantity of fresh milk.