Routers consume a really tiny amount of power, so you've no need to worry about any strain on your electricity bill.
Further, the exchange communicates with your router periodically (even if you're not online) so that the download speed can be optimised for your phone line. If your router is switched off the exchange registers a (non-existent) fault and drops the speed available to your line in order to compensate for it. So turning your router off is likely to reduce the download speeds you can get when it's switched on.
LEAVE IT ON!