Why not ask? instead of creating more false impressions? Clear it up like an adult, not a teenager.
You might be able to salvage the friendship. If you just blow him off, that's abusive too, unfortunately.
Or perhaps you don't want that possible rejection either? So to blow him off is sort of a preemptive strike---you can't reject me, because I reject you first.
And, novel idea here, if he asks for your number and you don't want to give it, say, 'sorry but no. I'm just wanting my single life right now', or something like that.