Grew up in a large Victorian house with big garden. Great for us kids but my mum hated the cleaning and we would have to help with cleaning and vacuuming. It was my dads family home and everyone, including me, started married life there. My mum was dads second wife and I dont think she really settled.
Eventually she got her way and they downsized to a little bungalow. My dad never got over losing his family house, the space, the big garden, even tho he didn't have the time or money to lavish on it. Also, whenever mum and dad had words there was no spare room to escape to. In his later years he was nearly always to be found in the greenhouse - cold permitting.
During his last months I took him back to have a look around the old area and familar places. We pulled up outside the house and both looked wistfully at it. It is now 2 flats.
I think of the old place quite a lot and memories of dad are usually of him there.
Your decision will be easier if it is just you making it. Couples must agree completely on when and where to move to. My husband was very nervous of leaving our house after 23 years (to a bungalow too) but he is settled and happy now in our new location.
I agree that you may have answered your own question. Good luck.