OP here. I first noticed this in my 20's. Posted about it in my mid-30's. Now I'm in my 40's. It's been more or less a continual thing.
Doesn't matter if I am having sex or if I am playing with yourself, spanking the monkey. It just matters that I have an ogasm and ejaculate.
It's like I am a battery and it uses up my battery life and then I need some time to recharge.
The downside of this is now my mental associations about sex are not completely positive. I still enjoy sex. But it's not all about fun and pleasure. In my head I know there is a price to pay. I know it will make me tired and if combined with other stressors like alcohol or lack of sleep I can actually get sick.
This has caused some minor problems in relationships when my female partner has wanted more sex than I was willing to provide.