It usually refers to the wearing of civilian clothes (I'm in mufti ...) by someone who usually wears a uniform. The origin of the term is thought to be from the Mufti, a Moslem cleric who interperets Moslem law, and used to wear a loose fitting robe and slippers, which officers in African countries probably wore when relaxing - hence the name.