"He may be right but that isn't the point..."
If you make a statement that is factually correct then it can't really be considered racist. A sweeping generalisation, on the other hand, certainly can be thought of as racist/ sexist/ whatever-ist. And nearly all such statements of a type "people in group x are more such-and-such than those in group not-x" are wrong, or at least very misleading.
In this case, it may well be that in Jamie Oliver's experience those staff who were born in Britain are, on average, less hard-working than immigrant workers, but there will always be exceptions. Background, too, is important: For an immigrant looking for work may well take what they can get and gladly, but someone from this country who ends up working in a kitchen might view it as a failure, or a temporary solution while they look for something better. In short: our expectations have risen faster than the opportunities have.