Hair loss and slow hair growth are signs of nutrient deficiency, particularly protein. I am sure your intestinal tract is damaged from infection and this damage does not allow nutrients to be absorbed.
Based on what I am reading this would likely be bacterial or protozoa infection and I would suspect yeast/fungal based on the white specks, usually yeast/fungal is secondary meaning the microbiota of the gastro tract becomes damaged which allows yeast to step in.
Usually there are bacteria that can cause problems in the gut but these are usually checked, something like c. difficile. You might want to ask your doctor if they can screen bad bacteria. It would be possible that a rogue bacteria is to blame, this would be a bacteria not normally screened for.
Protozoa are another issue since parasitic testing is known for its difficulty in finding these infections. Typical parasite type of cleanses that are promoted are not be going to be effective against protozoa. They don't respond to say Black-Walnut .
Intolerance to dairy is a sign of infection, this is called a secondary lactase deficiency which is common with people who have gastro infections. Sugar is a known promoter of many infections.
Fat intolerance would suggest that the liver/gallbladder are struggling. This is common with infections. Liver Flushes are not that effective IMO however products like liver tonics (usually items that contain milk thistle, dandelion and others) can help. The b vitamin choline may help as taurine or methionine. B vitamins help support the methylation process which will help the liver gain traction. It is important to have the liver healthy. I have learned through experimentation that if the gut is damaged then it will be more difficult to detoxify.
With infections in the gut then the gut wall becomes damaged due to the immune activation. Typically you see aberrations in stool formation such as loose stools, diarrhea etc. One of the big helps was using gelatin mixed with peppermint tea. The gelatin is something you can buy at the grocery store (Knox) or you can buy organic at Amazon. Gelatin has the same amino acid complex as the intestinal wall and will provide nutrients to rebuild it. Assuming it does not cause problems then 1 to 2 weeks should produce noticeable differences.
There are two paths you can go down
1/ testing route in which I would suggest ParaWellness labs.
2/ bombing route which means taking antiinfectives to force them out.