Sex is pleasurable, its fun to frolic. I'm an old fart pretty much and in all the sexy time I've had none has been for emotional attachment reasons, and never have I felt empty afterwards, well apart from the obvious. That does not mean that I am not emotionally attached to my wife of 30 years, it means that sex for me is a physical pleasure and not an emotional itch to be scratched. I suspect my wife sees sex as a renewing of emotional bonds in ways I do not.
In my youth I would really, really have to dislike someone to refuse sex with them. You see there is no good or bad reason to play hide the sausage, there is just the pleasure that if you do it right is shared equally.
There is a reason why the private parts are 'private' and why most people grow more attached to one other after having their first sex. It can be for fun but that is just, in my opinion, the wrong way of understanding an intimate activity that is suppose to make two souls linked into one.
It seems to me that if you have sex with another person just to satisfy your desires and not expect to develop a relationship with the other person or feel any emotional attraction to them, you are more likely to feel empty afterwards. If you're going to have a sexual relationship with someone, I feel it'll be more meaningful if you have emotional attraction towards him/her.