I am not saying you have a boundary dispute but if you did have a boundary dispute you may make some Solicitor very rich.
The title plan will not show a definitive boundary, often they are drawn at a scale of 1/1250 and in the absence of physical features it is very difficult to establish the exact boundary.
As you correctly and logically say you would expect the visible line to be the boundary, but this is not always the case as driveways are often built like walls just on the owners side in which case whoever had the driveway built would have sacrificed a little of their garden for the foundations.
On the other hand the driveway may have been built on the boundary and therefore by your neighbours own admission he has taken some of your garden for the foundation of the driveway.
Unless your neighbour has found a way to exactly establish the boundary, he will have the same difficulty as you. You will not want to get into a dispute with your neighbour, particularly if either of you is thinking of moving and the dispute would have to be declared. I suggest you negotiate with your neighbour then decide if you are to plant your flowers.
I had a case where the Local authority put in a sign just outside a property wall which the property owner claimed was still his property, the LA were unable to prove this was part of the road and removed the sign.