'Watch one's back, watch a person's back : be alert for, or guard oneself or another against, danger from behind or from some unexpected quarter' : Oxford English Dictionary.
Either the O.E.D is wrong or your brother is.You'd best ask him what he thinks.Perhaps he'd prefer 'Mind you back' .It has much the same meaning. After all, 'while shepherds watched their flocks by night' means the same as 'while shepherds minded their flocks by night' (but wouldn't fit the metre of the carol quite so well)
'Watch' does not just mean 'to look fixedly at' 'or ' to look at intently', as one might watch a play, it means 'to exercise protective vigilance over something ' or 'to pay attention to', 'to take care', among other meanings.