Check what you've been charged with. If it's an offence under Section 41D of the Road Traffic Act 1988 (as amended by the Road Safety Act 2006), then it relates specifically to that piece of legislation which bans the use of hand-held phones. (See here for details of the legislation:
http://www.thinkroadsafety.gov.uk/advice/mobil ephones.htm )
However, if the charge is under a different piece of legislation (such as Section 3 of the Road Traffic Act 1988, which deals with driving without due care and attention), you are simply charged that you weren't in proper control of your vehicle. Such a charge could be brought for anything which led to your attention being distracted, such as unwrapping a sweet, lighting a cigarette or dialing a number on a hands free phone. See here:
http://www.cps.gov.uk/legal/section9/chapter_b .html#21
Thus, if you're charged under Section 41D you should be able to successfully appeal on the grounds that your phone wasn't hand-held. Otherwise, you'd probably lose an appeal because the police officer would state that you weren't in full control of the vehicle.
Chris