Hi Coffee Lover -
The problem with coffee is that caffiene acts more like a drug in the body than a food. It really upsets blood sugar levels (among other things), which can cause a cascading effect of numerous other problems.
3-4x a week is still a pretty good amount. Especially if you are adding pasteurized milk and/or concentrated sweetener.
Getting it down to a "once-in-awhile" treat (1-2x month) would be of great benefit to your body. Try going off it completely for a good 3 months and see how you feel. Then add it back in. You will more than likely be very surprised to see how it affects you.