I've been there too. I always thought I had cancer or some other awful disease and that I wouldn't live long enough to see my kids grow up. Thank God that has gone away for the most part. I still get it every now and again when I feel something in my body that seems off, but it's so much better. I am about 5 months post removal.
I also seem to have irrational worries about family and friends having cancer or diseases too. I'll get so worried that I make myself nauseous. I'm sure part of that is from the Mirena, but it is also due to my mother passing away last year from cancer. That fear just won't go away.
Give it some time. The anxiety does get better although it comes in waves.