I think you should go and see his teacher.
Start by saying that the average four year old does not grasp the concept of a contract of behaviour like this, and explain in detail about the difficulties he has.
Always start from a spirit of co-operation and courtesy. No-one likes to feel under attack, and you need your son's teacher's co-operation to assist with this situation.
Hopefully an informal chat will sort this out, but if not, take your concers to the Head Teacher, with the same approach.