I would think it would be possible for a person to be allergic to almost anything, including ashwagandha. However, you might find it interesting to know that ashwagandha is actually a good herbal remedy for treating allergy symptoms...
"Ashwagandha. This herb is very popular in treating inflammation of all kinds, and allergy is always and inflammation. According to modern herbalism, Ashwagandha contains withaferin and withanolides, which are natural steroids and stablilize allergic reaction."
http://www.chinese-herbs.org/respiratory-allergies/
I also wanted to share with you that after months of suffering with insomnia due to adrenal fatigue, ashwagandha was the only thing that helped me to sleep. It was like a miracle for me!! I still continue to take 3 tablespoons every night.