As with all types of heating (especially electric) you must have a very high level of insulation. Preferably with a new-build. We'd need to know the age and construction of your house.
LPG has always been expensive, but really not a lot different from oil now. You might consider having an LPG bulk tank installed in the garden.
Dave's idea makes sense. An electric boiler. I can't think you'll make a huge saving on running costs though.
Jom mentions air source. I'm building a house right now in timber frame with a huge amount of insulation. No outer leaf of brickwork ... just the timber frame... USA/Scandinavian style. This will have an Air Source Heat Pump with underfloor heating (perfect combination.)
It is technically possible to connect this to your existing radiator system. The trouble is, it won't work terribly well because Air Source gives a temp of around 50 degrees. Boiler systems run at around 70 degrees.
This is Ok if all the rads are replaced with oversized ones to give greater surface area. More cost, but not insurmountable.
Modern storage heaters are a whole lot better than they used to be. They are only limited by the timing of electricity off-peak periods. If you can get a deal with a decent boost during the day, they are an option......... but a limited one.