I learned that calcium is antagonaist of magnesium but if that is true then it makes thing complicated with some mineral antagonist of others, it is hard too know how much is too musch with all this.

So I read for instance drinkinbg too much milk could hinder magnesium use.

Calcium is an antagonist to magnesium and can interfere with its use.  So excess calcium is an issue.  

As for milk though, milk is a very poor source of calcium.  Even though milk has a high level of calcium the protein in milk blocks the absorption of calcium, but the not the phosphorus that is also antagonistic to calcium.   This is one of the reasons milk actually increases bone loss.  This is a well know problem in the dairy industry, but of course a problem they do not discuss publicly.  But they try to counter this problem by adding vitamin D to the milk in an attempt to counter the calcium blocking effect of milk's protein.  But they use the inactive vitamin D₂ because it is cheaper than the inactive or active D₃s.  D₂ is basically worthless to the body though and has virtually no effect on calcium absorption.