You guys aren't alone. I have seen many reported cases of reactions to cacao. Some may be to mold or bacteria on it (hard to harvest) and some can be to one or more of the constituents that make it up.
There is a real chasm in the raw food community as to its value. SOme (that sell it) think its a super food and many others think its a stimulant that comes with potential problems such as mold/bacterial exposure.