I think it comes down to people that want to force their ideas (literally force them) on others. It just happens that many religious believers seem to hold a massive patent on this.
On the other hand, when's the last time you had someone come to your door and want to discuss infinite potential?