Oh, you know, I don't think that it is such a big problem, it doesn't seem he meant it to hurt you. If he has been living with her for 4 years, he just had developed some automatisms, but that doesn't mean that he misses the other woman.
In the other hand, try not to be jealous and accept that he has had this woman in his life before you, you can't take that away, but that's over now, so don't bother about it.