Right, but I don't have to eat paleo in order to eliminate refined carbohydrates and sugars from my diet. I guess your point is if you eat paleo, then you automatically eliminate all these from your diet. Okay, but then the problem is that someone that blindly follows paleo and is still clueless about nutrition could easily be lead down the wrong path. The big problem with paleo is that it is not one specific diet but a large collection of various diets all propagated by various paleo 'gurus' each of them very strict about following their own rules. Some of these diets can be quite healthy and allow for a balanced diet and others are quite restrictive that could easily lead to nutritional deficiencies very quickly.
In my opinion, instead of just following some random diet, the goal should be to become informed about nutrition and your own body. Then we can make proper decisions on our own without following some random diet guru who may or may not lead you astray.