Obviously this set up isn't making you happy anymore therefore you need to do something about it. You wanted to feel special and loved by him but as others have said he is just in it for the sex. He would not be upset if you ended it tomorrow, maybe he'd be annoyed about losing the sex but he wouldn't be upset about losing you.
You should stop this and go out and find someone who isn't already attached that will love you and make you feel special, he is never going to do that, he is never going to leave his wife and you are never going to have a future together, what do you want in the long run? To be a bit on the side for him to use or to have someone who you can have a proper relationship with? You're wasting your time with him.