I thought that when you dipped them into the egg yolk the yolk looked like the hats the Irish Guards wear, the ones with the big black furry hats and the red jackets outside Buckingham Palace (obviously not the same colour!!!)
The earliest recorded use of 'soldier' in this sense was actually to a piece of fried bread dipped in soup rather than toasted bread dipped in an egg. In both cases, however, the soldierly idea is contained in the rigid, stand-up-straight aspect of the bread.