Hi there, It sounds like you work for a smaller firm and maybe your boss is expecting of you what he would do himself perhaps?
Either way, you do not have an obligation to answer the phones but I would not hesitate personally, if nobody else was available - thats just being a team player really and preventing a potential customer form getting no answer - terrible for business!.
It may be that he is discouraging you from sitting at your desk while 'having lunch' as this can have the impact of making others around you ease off a bit perhaps? Just a thought.
Either way, I think it is a good career move to be the team player and a bad move to start quoting your human rights, etc, over something so minor and with reasonable viewpoints on both sides.
Hope that helps a bit, good luck