It could be that you and your brother just look different to your siblings. My Mum was one of 6 kids and 4 of them had dark brown to black hair, brown eyes and were inclined to be on the chubby side. My Mum and one of her brothers were both blond/e, to the point of almost being white, with grey eyes, and were very slim. It was always noted that my Mum and the brother she looked like took after my Grandads Mothers family, whereas the other kids obviously took after another side of the family. Oh - and there was 18 years difference between my Mum and this brother, and my Grandparents had moved to a different town by the time my Mum was born, so I think it highly unlikely that my Grandma had an affair which resulted in these two children!