Your story brings tears to my eyes and reminds me of why it is important to continue my research and writing on this subject. The fact that your Grandmother got better when she left the house is a strong indicator the house plays a significant role in her health problems.
I'll come right out and say it. Yes! I believe the house is the problem. All the health problems your relatives suffered from can be attributable to the mold problem in the home. If the building materials affected by the mold were not removed, the mold problem was not dealt with properly. Every health symptom you mentioned points to a toxic mold exposure.
Call in a reputable professional mold testing company to test the house. Be sure they send the samples in to a good lab, where the report tells you what species of mold, at what levels. Then you will have verification of the mold problem. This will explain all the symptoms you all have, and are, experiencing.