Why should your father just have to "Put up with it", as suggested by some people?
You've had confirmation from the surveyor that it's on his land, so yes, I'd pull it down.
He will probably call the police, but I doubt very much they will do anything about it, once he shows them the paperwork to confirm that it is indeed his land. His neighbor is obviously one of those people who just thinks he can get away with anything, whilst showing no regard for anyone else.
This would seriously annoy me!