You may already be aware of it, but there is a website call malesurvivor.org. You may be able to find a support group or one on one counselor in your area.
There are also books out there. I read one called Beyond Betrayal that really helped me realize there were other men out there and that I was not alone.
I know it may sound weird, but be grateful you have the memories of the events, this will really help you in healing process. I still cannot recall the actual events, but I know and feel in my heart that I was seriously violated. Just from feelings, tendencies, and reoccuring situations. It is a source of major anxiety and sometimes insanity to not be able to remember, but you know something happened.