nannon, I don't think you are a b*tch at all, and I can certainly sympathise with you. I am 21, my partner is 45. So we have often had the, "well, how different ARE we?" I think all relationships involve a degree of compromise, but at the end of the day you both need to be happy. You are wise to step out of your situation and apply some perspective.
I would suggest that if it is something you planned anyway, then moving to London won't be the upheaval that it could be if you are only moving to be near him.
I understand what you mean about what you want to do-my partner was in the army, and travelled wildly, so he's all done if you know what I mean. Whereas there are things I would still love to do. Now we have the dog, and I am at Uni anyway, I plan to travel later, but fortunately I can take the odd holiday, as he is happy to dog sit for me, and fairly laidback in his approach to life.
We had a really tough time over Xmas last year and the beginning of this year, but I don't think that is attributable to the age diff, any relationship can hit crisis point (and I am certainly not suggesting that yours will!)
I really do think you should give it a go, he might be all you've ever wanted and more! And if he isn't pressuring you, then there's no rushing into things you'd rather wait for. If you didn't stick with it, at least to see how it turns out, then I know from my POV I would regret not trying.
I wish you lots of luck and happiness, however things turn out. At the risk of sounding trite, you have a lot of time ahead of you to explore all the great opportunities life has to offer. x