Your boss is a nasty creep who is taking advantage of you and abusing his position of power.
You need to put an end to this behaviour. Believe me, he feels nothing for you, you are just a plaything, someone to act out his fantasies on, he obviously can't with his wife. He will only want to take things further and you will come out worse for it.
I doubt this will be the end of it for you and now you are in this situation, he sounds like the type who would make your life a misery if you don't do as he wants. I think you should start looking for another job, or if it's a large company you work for, ask to be posted to another department or branch.