A coach driver would be responsible for passengers in the hold, not for seated adult passengers who choose not to wear a seat belt.
You haven't been done for a seat belt offence. As you rightly say the responsibility for that would be your adult passenger, who would receive a fixed penalty notice.
You have been done for a Construction and Use offence, probably code CU50, Causing Danger By Reason Of Load Or Passengers, or CU60, Fail To Comply With Construct. And Use Regulations.
It is the driver's responsibility to ensure his passengers are sat where they should be.