i would test it at a low dose if you are concerned and then build up slowly to a normal dose. maybe start at 10% then if no problems 25%, then 50% etc.
i think allergies would be more likely if they are in the same plant family as Lavender, which i doubt.