Well, I graduated last summer with a degree in Physics with Astrophysics. Currently i'm doing some work in the Physics department I graduated in. Having said that i'm not entirely sure I want to continue with astronomy or the like. My degree was fascinating for the most part, if hard work, and now I have a working knowledge of most things astronomy related. A degree in Physics is not always that easy to find somewhere to work in. Not much in the way of jobs for physics related careers, but if you get this far you'll be able to go out and look for stuff yourself. There are jobs if time and effort are put into the search. People are always after competent scientists and mathematicians. Chris' above post is right about if you want a career in this area you'll need a good degree, then a postdoctorate qualification.
Before this you'll need to have good maths skills to get the grades to get on to the course (or be willing to work at them). I would absolutely say it is worth it mind. I loved doing physics, if not always the work load associated with it. And you'll always be able to tell your friends and family about how the space related stuff works, even if you only do physics until you get your degree.