You are indeed, JudgeJ. The monarch of the day, or a stand-in, presents such honours. The honours are awarded by the government, of which the Prime Minister is the head. :)
I just googled and it said Harold Wilson. I think John and Paul returned theirs, not sure about George and Ringo, but its more than likely that they did.
Perhaps we are only dealing in semantics, shammydodger, but I understand that the Queen chooses the recipients upon the advice of the Prime Minister. I would therefore say that the awards are ultimately within the Monarch's gift and not the Prime Minister's.