Is that the distilled water that comes in plastic bottles labelled not for drinking? If so, then I'm not surprised that it tastes horrid.
I've not had the pleasure (dubious or otherwise) of drinking distilled water from a potable source. I did consider buying a home water distiller but was advised not to worry about it until I'd made the really big changes in lifestyle and diet.
I have been using a water-filter as a best option so far and can only hope that it's removing most of the junk from my supply.