A 'podcast' is just a posh name for a radio programme (or similar content) available in MP3 format (rather than being streamed). If you've got an app on your computer/phone/tablet/whatever that can play MP3 music files (as you almost certainly have) you've already got everything you need to play podcasts.
For podcasts from the BBC see here:
http://www.bbc.co.uk/podcasts
(Scroll down to 'Find podcasts' and then select either the station or, possibly better, the category which interests you).