It seems to me that your boyfriend is using you as a sort of 'entertainment', rather than wanting to be in a genuine relationship with you. (i.e. he sees going out with you as akin to going to see a film. It's something that he enjoys doing but he doesn't want it to be a major part of his life). Either that or he's got something to hide, which he thinks his son or others might mention in conversation if you were to meet them.
I find it hard to see any real future in your relationship (or lack of one). In my opinion, the sooner you ditch him, the sooner you'll be able to move on with your life with someone new (and hopefully much better).