A few walls are actually astride the boundary. The best example is a link-detached house where the garage walls are clearly astride the boundary.
In your case you are saying that you own the wall. Technically of course he had no right to render it, but that is history, there is no point going back over that and it doesn't 'increase' his claim to ownership of any part of it.
The Party Wall Act would apply to you if you wanted to maintain or rebuild it, or to him in the even that he wished to build a wall or fence on his side immediately up against yours (unlikely, I realise).
He is of course, entitled to cut any part of your plant growing on the wall that pokes its way into his airspace - assuming that is where you regard the actual physical boundary to be. This is nothing to do with the Party Wall Act, of course.
Suggest you have a civil and calm conversation with him at an appropriate. Something alone the lines of - Hello Bert, please could we have a short chat about the wall? Then open up with 'How do you see the ownership of this wall?