It depends upon two things:
(a) are other local residets also putting out food?
(b) have the local birds come to rely on your garden as a major source of food at this time?
Just now the majority of birds have fledged a brood and some are on to a second brood. If your garden is a major food-source in your locality, and they have come to rely on it, then removing it now could be a disaster for current broods that cannot move on immediately to other areas.
I find that demand doesn't drop off until October.
At the moment I am not aware of any other neighbours putting out birdfood so they are very dependent upon what I put out. At the moment, they are getting through a kilo of mixed birdseed a day (The feeder is filled twice a day) + 5 or 6 fatballs and a small amount of peanuts (the peanut feeder is refilled once a week). If I were to stop feeding them right now, I'm sure it would cause stress to many local birds.
For many tears now, I have had food available all year round, and will continue to do so.