I would drive with the windows down all the time, except when you are in heavy traffic. Take back roads and such when you can.
I hate the "new car" scent and many car smells in general are too strong for me... I got a used car with leather seats, that combo seems to be the best for me, and I still drive with the windows down 90% of the time, unless it is extremely hot or cold (rare) or if I have passengers that complain. :-)