Having been a rider in the past and having come across good and bad riders as a driver, I can see both sides.
There are some very good, polite, riders out there (like the chap I passed a while back on a country lane, riding one horse and leading a very skittish youngster, who still managed to thank me when I slowed for them). There are, sadly, some very ignorant ones, too - such as the pair who insisted on riding two abreast even with a line of cars behind and a wide verge beside them.
But the plain fact is that there are becoming fewer and fewer places to ride horses safely off-road, or that don't involve getting to them via a road. There's a limit to how many hours can be spent cantering around the same paddock before you and your horse get bored out of your brain.
Everyone has a right to use the road in a legal manner (it's not just road tax that pays for them), and has the right to be safe whilst doing so. But with rights come responsibilities and so we should all take responsibility for our own safety and that of others. The law dictates, for instance, that pedestrians shouldn't use motorways. Anyone trying to cross a dual carriageway on foot must be a nutter. But we try our best (I hope) not to hit them in either case.
If some of the more ignorant riders bore this in mind when encountering other road users then perhaps the more ignorant drivers might be more inclined to give them a little more space. And that can work two ways.