Many plants of the Anacardiaceae family cause problems.Its the sap of the plant that causes irritation.
So processing the berry to extract the nut is the issue. As the shell of the seed contains Urushiol, which is the toxin that is in Poison Ivy. The fruit itself is edible.
What is sold as "Raw Cashews" are indeed cooked, but not Roasted. As they steam the seed to remove the toxic shell. However, they can be eaten Raw.
Pistacios and Mangoes are cousins to Cashew, but don't know whether there is issues with them...