Actually, I'd say pretty much the same as you have done, Taichi.
It has long been the case that nations blame their neighbours - especially those very close and against whom wars have been fought - for all sorts of failings. Thus, we make fun of �Dutch courage', which appears only if the Dutchman is drunk. Similarly, our slang for a condom includes the word �French' and theirs includes �English'. When we skive off, we take �French leave' and when they do, they take �English leave'.
�Pardon my French' is very much in the same mould. The suggestion is that there is no such thing as a �bad word' in English, so - if someone imagines they heard one - they must have misheard it as French...or "I was really speaking French!"