I think this little boy behaves like this becasue he knows he can get away with it.
It seems a shame not to see your partner when his boy is there as this is not a real solution. Can you ask your partner to find out about rules from mum at home. Where-ever possible the boy will be more settled and more happy if mum and dad look after him the same way. I bet mum has rules!!!
If that is not an option say to your partner before the next problem, that you like spending time with both of them, but if the son ever kicks out that he must discipline the boy some how and if he doesn't then you will will. And tell him what you want to do and say. I would suggest for that age, say some thing like "Kicking is very bad, you have really hurt my arm. Now you must say sorry to me." (and if he doesn't then put him on the naughty step or take away a treat (no biscuit /pudding later) or favourite toy. It would be best if discipline comes from dad when he is there, but you should agree with your partner that you can discipline too, when dad isn't there or that will create another problem. Your partners "happy time" theory wont work long term, his son will learn to wrap him round his little finger, which he is already doing. and the behaviour will get worse and worse. Kids need rules to feel safe and secure. And they need to learn acceptable behaviour otherwise he could become an outcast.
If you cant agree anything along these lines then fall back to not seeing him while his son is there. or u will be a punch bag.
Good luck with your partner. Maybe just ask him what he would like to do about the problem and say you will back him up. but that the son must be stopped from kicking out., or make to see the consequences of his actions.