Nothing wrong, just set up a friendly pattern

Be more casual.

Once in a while talk about something you are going to do, and invite him. Give him room to say no, without embarrassment.

Have confidence. He's given you space in his home, whatever the arrangement. Just as you have given him space in your home.

Now learn how to be GUESTS in each other's space.

You will both like it.