I purchase Brazil Nuts at Wild Oats or Sprouts. I have purchased at Whole Foods, however Wild Oats seemed to have the better quality which surprised me.
I believe Brazil Nuts to be a little odd tasting as far as nuts go. I cannot help you with the mold... I think that I may have a high tolerance for molds as my primary weekness with food is cheese, especially good blue cheeses.