If your mp3 player only plays mp3 files, then yes you'll have to convert. It may support aac files too though (most phones do, for example).
If you want to convert:
in itunes, goto edit->preferences, then to the advanced tab, then to importing. choose to import as mp3 instead of aac, and press ok.
Now back on your iTunes library, highlight all your songs (click one and press ctrl+A for select all), and right-click one and choose 'convert selection to mp3'. then they'll all be converted. it'll take some time, and just leave your computer to work away.
if you want to remove your old aac files afterwards, right-click one of the column titles in iTunes (such as 'name', 'artist', etc.), and tick 'kind' in the menu that appears. now it'll show you which songs are mp3 and which are aac. click the 'kind' title if you want and it'll order them for you. then just highlight the aac ones and press the delete key.