It is Greek for 'the many'. That is, it means the common herd, as it were...the vulgar mob. For some reason - perhaps because 'hoi' sounds rather like 'high' - some people think it means 'the high and mighty', the exact opposite of its actual meaning!
Qm's answer shows why it is incorrect to refer to a group of people as The Hoi Poloi as the Hoi part means The so you are in effect saying The The Many! [bit like an ATM machine as mentioned elsewhere on this site]