If you take F1 which physically is, believe it or not, one of the most demanding sports then good and successful drivers tend to retire slightly later than most other sports.
I myself at 46, now have a 34 in waist rather than a 30 in waist but my BMI is still only 24 and I am as fit and strong as ever I was 20 years ago. However I found the amount of excercise required to prevent more weight gain and dietry modifications have increased as I got older to maintain the same level of fitness, but yeah wardy I agree, standing in a goal mouth some men could do well at 50.
We live in a youth worshipping age though where anyone over 25 is considered obsolete, so I think perception is a lot to do with it rather necessarily than reality.
There was a very indignant 20 something who lost with ultimate bad grace to me at squash and I'm sure he'd have lost to a younger man far more gracefully.
The problem is all in perception and as we live longer as a society we will begin to see two distinct groups emerge I think, of younger people who feel threatened by older fit capable people.