Your 'contract' comprises (potentially) lots of small elements - some written down and some not (what are termed 'custom-and-practice'). But I would struggle to believe your employer claiming that this requirement was a c-and-p T&C.
What I'm saying is that your 'contract' isn't just the letter of engagement you (should have) had - it also comprises all the other correspondence the employer has written to you or made generally available from time-to-time. Something like this I would expect to find in the company handbook (if there is one), because I will guarantee that stuff in any company handbook does form part of your T&Cs.
It is very common for an employer to make such a provision on 2nd jobs, because they don't want an employee getting dog-tired by doing multiple jobs. The fact that your job is part-time generally doesn't make a difference, though if you are doing so few hours you could seek to claim that it was an unfair term.
So I would check your various bits of correspondence, and if you can't find anything then ask your employer nicely where precisely this T&C is covered because you can't find it. See what they come back with and post again.