I hear ya. I used to be totally the same way. A few weeks ago I realized that oatmeal has gluten in it too, and I ate it every morning for a year lol.
I think it is pretty basic. Gluten protein is ALWAYS bad for the human immune system, it is just that it does not affect others at all because their immune system is ready to fight it. Just because they have no bad effects from gluten, it doesn't mean it is good to eat it.
I have been gluten free for the last 2 weeks, and I really have to say that I have already noticed a difference. Next up on things to avoid or not: Dairy.