Time can sometimes have a way of blurring and softening our memories. So, it's understandable that you didn't "remember stuff" right away.
Without knowing all the details, I'm thinking that either your ex's brain damage could be responsible for his behavior to you or he is, as you say, looking for his next victim. If it is the latter, no doubt he is looking to soften you up in the hopes that you will take him back and take care of him. If so, don't fall for it.