They would, in the earliest photos, have to sit or stand in the same position for quite a long time in order for the photographic plate to be exposed to the light. The head and neck would often have been supported in a brace to keep it still.
Would you want to smile in those conditions?
On the other hand, it would have been a very serious business and a very prestigious thing to do. You'd have wanted to look important and wealthy, and important, wealthy victorians didn't smile in portraits.