Your panic attacks should improve if you are allergic to something and stop eating that product.
Here is why:
All reactions in our bodies are chemical. Your emotions are neuro-chemical. Panic attacks are caused by chemicals surging through your body.
When you consume something your body cannot process, you are having improper chemical reactions. Basically, when you eat wheat - your body chemicals (ALL OF THEM) are outta whack!
So yes, your panic attacks should improve. Panic attacks are fear. When I stopped eating wheat (I'm a type O - allergic to wheat) this crazy constant emotion of fear I had always had stopped. I'm not scared anymore.
For some reason I find that fear, panic, etc. is common with food allergens.
Give yourself 1-3 months. And allow your mind to process the change. :)