I believe...

that the toxic substance thats usually referred is a cyanidic compound found in the pits/seeds/stones of various Rosaceae fruits - this is why you don't really eat Bitter Almond, even the Edible Apricot kernels are a specific variety.

If I remember rightly, the compound is found in Amygdalin/Laetrile {Vit. B17}, due to the Prussic Acid, its benefit was allowing things to dissociate in the blood. It is necessary, but only TRACE amounts - people have supposedly died from eating too many of these 'stones'.

But to answer the question, the usual way to peel the almonds after soaking or Blanching is to rub in a linen or towel. This is usually done for culinary puroses as in desserts the skins can slightly tinge the flavour or darken the icing...