I think he meant solitude in a way that he can still be himself and have his 'space' even though spending time with this certain person. Atleast thats how i interpreted it. Now if you want to be 'left alone' all the time, then it will be somewhat hard yes i agree.