Just avoid milk and milk products. It may be difficult for you, but necessary. You also may be allergic to other things as well, but start with the milk.
There is a pulse test to use to find out which foods you are allergic to. If you google "pulse test allergies," you will find what you need.