Time to add another bit ...
Were we thinking about our spouses at home ? Yes and no.
At the point where things went past friendship, the point where we both knew that one day we were likely to get round to sex, one rule was agreed. What happened between us was between us, and if there was any chance that it would affect our marriages we would end it.
At that time we both needed a close, intimate (not in the sexual sense) relationship which we were not getting at home, but through it all our spouses were, and still are, at the core of our lives. Of course there were times when we were not thinking of our spouses, but that doesn't mean we forgot them entirely !
Did you ever see an old film called "Same Time Next Year" ? It covers a period of 20+ years, where two people, both married, meet for a week once a year. They are often out of step, have their ups and downs, but in the end give each other a level of love and support that they don't get elsewhere. That's how it was in my case - except that we met a bit more often than once a year.