I think that almost everyone has the same problem as you, at least everyone over the age of 55 years.
Indoor acoustics for TV programs can be " contrived" for maximum sound perception, height of ceiling, consistency of walls etc, but outside, the elements are on the driving seat and cannot be controlled.
So yes, it may not be your fault.
However if you have difficulty in hearing in other situations e.g having difficulty on what other people are saying, then see your GP and have a hearing test.