My women presently has a panel needing replacing in her fence, which by some amazing coincidence is the panel a past renting neighbours' kid used to hit often with his football. Strange that. After all the fence panel should have been strong enough to take anything the neighbours could throw at it plus more.
It is, at best, bad manners. Maybe have a polite word with your neighbours saying you are concerned and asking if they could move their carpet. Best to convince/persuade rather than act as catalyst for bad relations.