I would definitely stop the Fluconazole (Diflucan) if you are certain you contend with Asper. A bevy of natural antifungals are effective:
clove
cinnamon
thyme
rosemary
eucalyptus
lemon
sweet orange
lavender
GSE
pau d' arco
oil of oregano
undecenoic acid
coconut oil
cayenne
turmeric
zinc
C
to name but a few.
I believe that of all naturals, oils are most potent against Aspergillus, due to its typical colonization sites. The first seven or so of the above are good ingredients for a Thieves oil blend. I make my own, but site below is incredibly informative.
See Mold/Candida/Fungus Studies for abstracts on specific organism/specific essential oil efficacy.