Sorry to sound harsh but I don't care what anyone says - size **DOES** matter - and I'm talking from experience.
It's a harsh fact of reality. Nature can be unfair.
It can also be exhausting *not admitting* this fact when in a long-term r/ship and constantly reassuring the man in question that it's OK, when you both know it's not. Try doing that for years.