I would try the mineral oil anyway, it does no harm at all, and you will know within a day or two if kitty feels better. Mineral oil is only a couple of dollars at the supermarket.
Mine had the same: some few little specks, but she was scratching a lot. Now she is just fine, I only used the oil a couple of times.
However, it may turn out to be allergies as Ynaig suggests, or an ear infection.
For the infection my kitty was prescribed an Iodine solution, but don't remember the name. Infection will cause pain - so that may explain kitty being skittish about being touched.