Perhaps it's the pasturization process that makes them not bother you. Heat alters protein and many other elements in the almond. While the process destroys the nutrients most people want from their almonds, it might also destroy what you are allergic to. That's possibly why almond milk doesn't bother you.