The basic question is this - How long of an exposure to any heavy metal is unhealthy? It can take many hours for the body to purge itself of Barium and Gadolinium - if all of it can ever be removed from the body at all.
Gadolinium might be equally sinister it invades the brain by crossing the blood/brain barrier. This must happen to penetrate the lesions of the brain so they will show up on MRI images. Otherwise, MS lesions will not appear on the scan.
This entire issue of contrast agents (so named by the medical profession) is a very difficult one. Most likely, no contrast agent exists which will be broken down by the body, work effectively and all the while be completely non-toxic. Both X-Ray and MRI technologies must have these agents to make otherwise invisible structures appear on images.
So the question remains of contrast agents when is too much harmful, to whom and when? It could be years before we will hear anything on this subject from watch dog agencies, if ever.