Confused

I'm all over the place with my feelings, sometimes I enjoy having him here and sometimes he works my nerve. What to do? He's my friend but right now he is working my nerves. The bad thing about it is he's really not doing any one thing its just every thing. Have you ever felt like that about someone? Every thing they do works your nerves. I feel bad about it cause we asked him to stay but I was having second thoughts because I knew I wanted to be in my home with just my family. What do I do? I want him gone but I don't want to put him out. I just want him to find him a place to live. Is it wrong to just want to be with my family? I thought i was being a blessing by allowing him to come stay but now I feel like I'm being punished cause he gets on my nerves. *sigh*.