As one who ENJOYS swallowing because my SO tastes so wonderful, and as one whose SO does NOT pull away and use his fingers when I'm having the big O, I would have to say YES, I would wonder big time what was wrong if he did pull away and use his fingers. Just as he would wonder what was wrong if I didn't swallow.
But, let me say, I wouldn't want to swallow if my SO was taking prescription drugs or ate meat - because then he wouldn't taste as good to me (as I said in my post above, my SO and I are both vegan vegetarians, both eating a ton of fruit - which results if both of us tasting "sweet" to each other).