It all depends on how you go about it. To make a trip specially
to the office to give him your number, while dressing any differently than you were the first time... if it were me, I would think you were interested in one of two things.
Now, if you dress casually, as I assume you were the first time you met, and you happen to be in the neighborhood, and I mean really just happen to be there... not just using an over used line, then sure. I say go for it.
How the receptionist knew he was dating but not seriously... people talk. Especially in a close work enviroment.