With all due respect allow me to make the following observation.
It seems to me that you have already "diagnosed" him and may not realize that we are all capable of change... including your husband. Love can transform. You may need to give him a chance to change and live up to his outside "reputation" of being a good, caring father. If you allowed him to be with his child (in short time periods at first) and learnt to care for someone other than himself, this may be possible. I´ve seen it happen with my ex-, who eventually became a great dad... in spite of our divorce.
Nobody is perfect, I´m sure you too have your shortcomings.... we all need a chance to learn... that´s what life is all about IMHO!
Give Peace / Love a chance ;)