I know I'm being an idiot but how can I stop feeling like this?

I am having the stupidest feelings about my current partner. He's just come out of a marriage and his life has changed incredibly and I also separated from my husband last year. My partner and I live in separate houses but see each other a lot. Here is the 'problem'. Sometimes he wants to stay at his house for a night (or two) at a time, or just spend some time during the day to chill out, but everytime he says he is going home I get this horrible sinking feeling, regardless of how long he has been at mine or where we've been or what we've done. We have an amazing relationship in every way, but I'm scared my stupid feelings are going to push him away. We talked about it today and he said that he just needs time to breathe sometime, and do his own thing, and he likes being by himself sometimes, and because he has just come out of a stifling relationship he needs it. He said he appreciated me being honest but my feelings are unhealthy (and I definitely know this!) but I don't know how to control them. Why is it that even though I understand his reasons, and know that it is healthy, I still seem to have this need to be with him all the time, even though there is a part of me that enjoys being by myself too? Why do I panic that he isnt coming back even though he always does now. Our relationship didn't have an easy start and there has been reasons to mistrust him in the past but we've worked through that. I am struggling with his fiercely independent nature because my ex-husband was clingy and that was all I am used to for the past 14 years. I just need to know how to calm down because there is no basis for this paranoid and possessive feeling!