Sounds like it's been more traumatic for you than it's been for him. I would pick a time when conversation turns intimate and I would pour out to him what happened and how I still feel about it. Then I'd tell him how very sorry I am now and above all, how much I love him. This would give you a chance to get it out of your chest, hear the forgiveness coming from him and thus allow you the opportunity to forgive yourself as well.
You never know, in some very deep layer in his soul, he may also have some pain from the experiences, although being so little, he probably doesn't remember.
Healing needs to happen and can't happen while you keep harboring it inside, but love opens the door for all kinds of wonderful things.