Well, you are getting lots of different opinions, so maybe I'll add mine:
Your nature doctor had no business prescribing ashwagandha and ginseng, for heaven's sake. They are too strong for a weakened and toxic body. You have to do a bit of detox first, then maybe slowly re-try those herbs. Maybe. Look this all up in Chinese trad. medicine for an explanation. And best to you! Keep trying.
P.S. Look into niacin (the flush free kind, inositol hexanictinate)for any mental/ Depression issues. Stress B complex wouldn't be strong enough.