Eddie, I'm aware you know a fair bit about these things (as you often contribute to others' questions).
The law and hence an Employment Tribunal differentiates between a 'worker', an 'employee' and a 'self-employed person. Both 'workers' and 'employees' benefit from having basic employment rights - covering for example, NMW and rights to prevent one being exploited.
Both 'casual contracts' and 'zero hours contracts' are examples of agreements that would normally constitute you being a worker, but without the wider benefits often provided to 'employees' of the organisation - sick pay, for example, except the right to Statutory Sick - which is part of basic employment rights.
I can't see that you'd be any worse off - indeed you may be better off because a zero hours contract will invariably mean that your contract is with the organisation, not the agency, who of course currently take a cut in exchange for dealing with your payroll admin. You may be able to agree that some or all of this cut comes to you.