It sounds to me, as though you have become his favourite Booty Call. He obviously likes you as a person, seeing as you were in a relationship previously, but he's now found it works well for him with you, purely for sex.
Where you are going wrong and will probably end up getting hurt, is that you're giving into him wanting to see you, in the hope that it means more than sex to him, it obviously means more to you.
If you're not happy to be just a F**k Buddy, then you should stop what you are doing or at the very least try to find out what it is he wants.