Mainly midges like stagnent water to lay their eggs in, so if you've got a pond that will attract them, a rain barrel they like them, what about Fir trees they like them too.
Get rid of all of those & you should be midge free.
If your neighbours have these the problem will be with them unfortunately.
(Midges fly over fences)
Stop scratching.....