Hi Hedge. My daughter is 9, and has been having the same issue since she was 4. In our last house, we never felt happy, and she regularly said she saw two men, who didn't want us there. From her description, we think they were dressed in victorian clothes. When we moved to our current house, she told us, they hadn't come with us, as they were stuck in our old house. In this house, she has told us, there is a little girl, but she is happy for us to be here, and just wants to play. Our son, is only 8months old, but sometimes appears to smile at something we can't see. We remain open minded, and alway accept what our daughter tells us....after all, just because we can't see something, it doesn't mean it isn't there. The only thing I can suggest other than what you have already been told, is going to sound a little silly.....but next time your son sees someone, ask them together to leave your son alone, explain that he is scared and would like to be left alone. It does feel silly, but it may help. Don't ever tell your son there is nothing there, as it will just confuse him.
Stay open yourself, you never know! Hope I've made sense, feel free to ask anything as it can help to know there are other people experiencing the same thing.