I gave a simple answer to the question as posed.
As andy-hughes said, when it starts there are only two people involved, and that is the point at which they decide if the affair is acceptable or excusable. I imagine that most people, looking at it from outside, would say is is not acceptable. But for as long as the affair only involves two people, what others might think is irrelevant.
But if it bothers you, how about this for a conversation, overheard in the street last summer, between two women :
"I got back from holiday with the kids and he's moved out and set up with someone else. He won't even see or talk to his kid"
"Oh well, never mind, you'll soon find another one"
"Too right I will. I'm going out at the weekend to see who I can pick up"