i dont think you understood your doctor correctly...

if someone tests positive they have it. they may not have an outbreak. but they have it in thier body somewhere and could pass it to you with skin to skin contact.
doesnt mean that will be what happens but it could. i have been with the same person for 6 years and never passed it to them.

if you did catch it from him it is possible that you have it without symptoms. and that is good because it means you are very healthy. if you ever have an active open sore you can go get that swabbed. there are other blood tests you can take to see if you have herpes. it doesnt refer to location of infection. just that you have antibodies in your system that are picked up by the test. this means you can pass it on to others.