how to set limits and handling inner fear.

Hi to everybody.This is my first post and I just feel very confused about this issue and need some advice.My boyfriend and I have a generally good relationship but sometimes he gets so irritated with evrything I do(how I drive,Because I lose the keys and only find them after a while,because i am difficult with food and always complaining about feeling ill-I have yeast overgrowth and IBS-etc)when he is ok ,he doesn't mind but when he is irritated he says I am ennerving him.We don't live together and we decided to spend the week without seeing each other and we just spent the evening together watching a DVD and then he told me that he feels depressed and irritated even when he is alone on one hand but also better while alone because he is doing some meditation practice.But he also said he didn't miss me and then I felt hurt and there was tense silence,and he said "well just want to be honest".I don't know wht is going on,but I feel scared deep inside,about him not loving me and also get blocked with a subtle inner fear when he gets irritated and try to be patient and do everything right to sooth things.However,the more I try the worse it gets and I am afraid that the situation ends up like one woman frien's of mine whose husband just got irritated by her way of eating a biscuit.What attitude shall I take ,I feel so confused