Any tonal language is going to be exceptionally difficult to learn for someone who normally only uses prototypical (non-tonal) languages.
So obviously the two Chinese languages, Mandarin and Cantonese, tend to feature high on the list of 'hard to learn' languages. However Vietnamese and Thai are possibly equally difficult, along with other tonal languages such as some of those spoken in Africa (e.g. Igbo, Yòrúba and Zulu) or among North-American tribal groups (such as Navajo).
Even languages that generally only use two tones (such as Japanese, Swedish and Norwegian) can prove very difficult to master, as can some tricky protypical languages (such as Finnish).
Some Slavic languages, especially Polish, can easily confuse learners because of the many different endings that each word can have (depending upon where and how the word is used in a sentence). In Polish, for example, the endings of proper names keep changing. So the US fast food chain isn't simply 'McDonald's' in Poland. It can be, among other things, 'McDonaldsem' or 'McDonaldski', depending upon where it occurs in a sentence. (I spent a year trying to learn Polish. It's hard!)