Fruit has Sugar (fructose) in it, so it will cause a yeast reaction. A small amount may not cause a reaction which is noticeable (it depends on how sensitive you are), but if you continually break your diet, or break it with something which has a lot of Sugar in it, then you will notice it.
There's some debate about whether or not lemons, grapefruits and cranberries are OK.
You might also consider whether or not there is something else causing your yeast problem. For me I believe it is heavy metal toxicity (in my case it is mercury from the silver fillings I used to have). I'm not yet completely cured, but chelating has reduced my yeast symptoms from fairly severe to relatively mild.