As long as you are not ingesting a really large amount, e.g. 10+ grams, of ascorbic acid SIMULTANEOUSLY with the milk, I would not think you would have a problem. But to be safe, you might want to start by ingesting smaller amounts to make sure there is no unpleasant reaction, e.g. I once came in thirsty from working out, and drank a quart of milk followed by a quart of orange juice [all probably within about 20 minutes] and had SEVERE gas pains in my stomach for several hours. Acids and alkalis IMMEDIATELY react to produce gas!
I hope that helps,
David
P.S. And you are correct, i.e. milk does have a [very] small amount of ascorbate in it.