Mattk: first, make sure your upload bandwidth is not being used by anything else. BitTorrent gives more download bandwidth to those who are sharing more. What is your upload and download bandwidth? In azureus, you can set how much you want to use: you should be using good settings..
http://azureus.aelitis.com/wiki/index.php/Good _settings
The torrent itself can also play a large part. Make sure there are plenty of seeders and peers, and that the torrent isn't restricted somehow (some are for members only, but will let non-members download at low speeds).
As a test to see what your system can manage, try downloading the open office torrent:
http://distribution.openoffice.org/p2p/index.h tml
With that download, I get download speeds of about 230kbps.