Music is such a subjective thing, and so diverse, that I think you can only theorize with this question; there's probably no one correct answer.
My theory, though, is that it's because music has a certain order to it; crap music is stuff that isn't disciplined, it's out of time, notes jump all over the place, and sounds don't fit any points on any scales. Even abrasive stuff like metal and punk has to obey these rules. Humans naturally love order, and organisation, so music represents the easiest way to create order in a universe otherwise full of chaos, and to stamp your own identity on a part of it.
Sorry if that's pretentious, or contentious, but it's the best I could do.