If you're talking about iPlayer then I don't think it has much to do with file size or speed, download speed from the BBC is much less than than the 50Mb that my connection regularly tests at, I assume that the pinch point is at the BBC's servers, I'd guess that they give priority to those streaming rather than downloading. That said, I haven't noticed any better speeds when downloading at 6am when I would expect relatively few to be using the service.
Once took 36 hours to download a 1.5 hour programme but that's Norfolk for you! It's been much better recently, someone's put a couple of pounds into the meter I think.