In my past, I reacted this way to anything that I was allergic to. I never had a panic attack until I used feverfew and garlic, among other things I was allergic too. I also only have heart palpatations when I have eaten something I am allergic to.
You might consider starting with some liver cleanses, so that your allergic response is lessened.