S100 from your own addmission your only 18. Lack of trust is one of the biggest reasons couples break up. You have a long life ahead of you and no doubt a few more boyfriends before you find 'the one'. I know the one your with feels special, but jealously only brings arguments and unhappiness. I think a lot of female ABers and maybe male as well . Have experienced very similar romances and have been torn apart by making the right decision.
Yes, it will push you away from him, it already is, which is why you have posted this question. It is always hard to know the right time to break of a relationship, but the longer you leave it the worse it is and more the likelihood that it will end with strong resentment. What ever your decision, I wish you luck and happiness. xx