enzyme inhibitor

I think the 'toxin' idea you had, may have been in reference to the enzyme inhibitor
in the skin-- but this would be in the raw, dry almond-- hence, the reason for soaking. it's easy to see how that information could translate to an idea that one ought to remove the skin. I occasionally have, only for that aesthetic mentioned: the taste of a soaked skinned almond can be sort of ambrosial. Usually I prefer the whole food though.

Even if I wanted a very white milk, I'd still blend the whole almonds, and then strain and use the pulp elsewhere.