Much as we Suffolk people hate to admit it, Norwich is a superb tourist destination. It's got a vibrant might-life, with several excellent theatres, loads of antique centres for browsing around, tons of history, an excellent covered market, superb pubs and restaurants and just about everything else you might look for in a short-break destination.
So I'll give my backing to Rockrose's suggestion.
(Living just down the road from Bury St Edmunds, which is a lovely town, I'd be happy to recommend it as a base to explore from but, in itself, it doesn't offer anywhere near as much as Norwich does. I've never found much to interest me in Ely though; there's not a great deal there other than the cathedral).
PS: If you do decide on Norwich, please post again and I'll be happy to point you towards some of its hidden gems.