Being self employed in one capacity (with Avon, which is fine), has absolutely no relevance to whether the other job is self employed or not. It's not in the same line of business and the Avon thing would rightly be ignored by the Revenue in deciding whether or not you are self employed in the other job. I have to say on balance from what you've said so far that I think you are employed and I expect the revenue would too. You don't do book-keeping / pa work for anyone else, haven't suggested that you have tried to find other clients to do it for and he sets your pay rate. Even if you do decide when you do it that's employment for me, not self employment. Do you get holiday pay or sick pay?
You are right though, the main problems that creates are his, not yours. He would have employers NIC to pay and if he isn't already doing so, holiday and sick pay too. From your point of view there are some advantages with cash flow to being self employed and also the things you can claim against tax are much more generous to the self employed but then as you are self employed in another capacity you wouldn;t lose them anyway. I'd say it would be in your interest to become an employee but if he'd rather dispense with you than do that you are pretty stuck.