the idea is that the natural protective mechanisms of the fruit is intact, that the item is alive still. rot and vermin only attack dead things.
what you say is true of normal fruits and veggies. if the chemicals dont kill a worm, then hopefully they wont hurt you either. this we know is not true, because the chemicals bio-accumulate.