One reason that men tend to be more prone to these types of infestation is due to door handles in public restrooms. Many restroom doors open into the restroom and the doorknob or handle must be grabbed and possibly turned. Even if you had washed your hands, if the person before you had not, you may pick up nasty things.
I always try to use a paper towel but some restrooms only have blow-dryers.
If the door opens outward, you can push it with your foot or even butt it open, resulting in less risk of contamination.
Funny how few people think of this.
Can't speak of the ladies room as I make no claim to being gender impaired or trans-gender but I assume that they are about the same. Ladies, however, tend to clean better then some men.