No, it's not illegal, but it has been shown that you have less control over the pedals which have been designed to work with shoes, and if caught, you could be charged with driving without due care and not being in a position of having proper control over a vehicle.