When I ran a rail station, the race day policy was to ensure that the relevant trains had the maximum number of carriages but not to provide extra trains. (Most train operators don't have the rolling stock or train crew for additional services). If the system seemed to be creaking under the strain, I'd book coaches to take people directly to the racecourse for the same fare as they'd normally pay to the rail station.
Looking at a rail route map suggests that travelling via Worcester is the shortest route but the National Rail website sends you via Birmingham most of the time. (I suspect that this is because the Oxford to Worcester train service is a slow route, whereas Birmingham services are faster). The Saver Return fare is �28, plus a �4 supplement for the coach to and from the racecourse.
The cheapest route (although it might get you there a little later than you require) is to travel out at 1055, via Didcot Parkway and Swindon, arriving at Cheltenham Spa at 1303. You can book an advance purchase single fare for �8. Giving yourself time to get a meal and a drink before you come back, you can take the 2202 service ex Cheltenham Spa, changing at Reading and arriving at Oxford at 2223, for an advance purchase single fare of just �6.
Rail times and fare taken from here:
http://www.nationalrail.co.uk/
All public transport from Oxford should be on the Traveline South-East route planner.
http://www.travelinesoutheast.org.uk/se/XSLT_T RIP_REQUEST2?language=en&timeOffset=15
Since Cheltenham is in the Traveline South-West area, it might be worth checking their website as well:
http://www.travelinesw.com