Hippyhoppy, I think you need to take legal advice. I think there is someone else and yes they would put up with it, because men are full of charm and he will have convinced her its for the future good and all that.
For him to allow you to come into his bed on hol, he's cruel. Playing with your head. Also just taking the path of least resistance, it's easier saying yes than it is to say no, what sort of mood would you have been in then whilst on holiday.
I don't believe he can just come round when he feels like it, and I think you might find that he has to keep a roof over your head until your youngest is 19.
However, if your going to start a fight make sure he has nothing on you. You need to be a sensible, model mother.
I would also get a private detective, they are not that expensive, just in the evenings for a month or so. they are usually very good I did it to my husband, he was doing it with someone from work, turned out he'd been doing it for years.