I think he's given you a hint of how life with him is going to be by dicussing his former relationship with you. Are you ready to live like that?
I don't think you are being selfish necessarily. I think it is healthy that you are thinking long and hard and realize that it may be wise not to commit to the marriage. This is a huge change that will be your life should you choose to move forward with it. And once in it if you are not happy with the arrangements he will certainly be able to say that you were forewarned.
This may be a phase he is going through...to recapture his youth...but then it may not be. Were I you I would postpone the wedding and let him get into his passion and see if you are willing to sign up for it long term. His response will also tell you where you fit in on his priority list. You have real concerns here that an "oh honey don't worry" won't necessarily satisfy.
Take your time. It is YOUR life too and you need to be confident in your place in this arrangement.