Get on to the taxi operator to complain and if they don't deal with it get on to the licencing authority. You don't pay a taxi fare to sit around whilst the driver faffs about.
It has never happened to me, though I did have a journey where the driver spent the entire trip arguing with somebody on his hands free loudspeaker phone. He did not acknowledge our presence at all, didn't ask where we were going (though I'd booked and paid via an app, so it was logged anyway), didn't check we were OK before moving off and continued to rant at our destination whilst holding out his hand for a tip. I gave him two - don't get involved in heated arguments whilst you're driving passengers around and acknowledge their presence and check that they are securely seated before driving off.
I contacted his operator who clearly didn't want to upset him so I got on to the licencing authority. They took a statement from me, interviewed him and gave him some "advice". He turned up a few weeks later to pick me up again. I sent him away and used another company.