It could definitely be the xanax. I am on klonopin and it has ruined my life. I started with Ativan for 5 years and then I switched to klonopin. .5 used to get me through the day but now my body needs more and I'm not giving it more so I'm having all of these mental problems (short-term memory loss, DP/DR, panic attacks, etc). Benzos are only supposed to be used for a few weeks, not years.
I'm seeing a psychiatrist on Friday to discuss my options. It's going to be hell, but I know when my body is free of this drug things will improve. Maybe the same thing is happening to you. Your body wants more and you're not giving it more so you're having all of these withdrawal symptoms.