I'd agree, but that said, sperm are incrediblly persistant little sods, because nature made them to reproduce, so they have a good go at it!
As long as his semen didn;t reach your vagina, and neither of you handled your vagina with semen on your fingers, you should be OK, but 'heavy petting' as it quaintly used to be called in my younger days is a risky bsuiness.
As the woman, you are the one who will get pregnant, so you must remain in control of the situation, even in the heights of passion.
Sexist? No, just realistic. It's the way we are made, and that's it.