What if the dr didn't remove all the amalgam fillings?

Hi,

I have 2 Amalgam fillings, and I replaced those like 3 times before and even swallowed some of it :(. Drs said "it's ok, it's harmless". Anyway, I want replace them with the white fillings which don't contain any mercury. But what if the dr didn't remove all the alalgam fillings.

What if he removed say 80% of it, by mistake, and put the new non-mercuy fillings above it?
Would it still hurt me? I mean cause my mercury levels to be high?