Can only make wild guesses. It seems unlikely ethernet could affect broadband to me as I'd expect the board to have processing power such that it sorts out the ethernet without using significant common PC resource. Although I guess if that is not the case there may be a bottleneck for some resource somewhere. Processing power maybe ?
I believe the shown connection speed is what ethernet is connecing at, your throughput of data will be less. 20 sounds a bit off though. Is there a problem with your ethernet card ?