Yes I agree with ward-minter - I have been told that exfoliation is the answer. Use a body scrub or sponge every time you shower and it should keep them away. Thats if youre a girl of course.
Exfoliate and moisturise, this problem is often worse with waxing I find! There is a product called Tendskin, which you can dab on with damp cotton wool, it encourages the hair to pop out.
Or if not Tendskin, then Ingrow Go by Skin Doctors.
The hairs are not strong enough to push+grow through the skin which is why they have become ingrown. Removing dead surface skin by exfoliation will help. But shaving is kind of a way to exfoliate I think as it's so close to the skin, perhaps you could try shaving more often. Ingrown hairs commonly occur when the hair is removed at the root, for e.g. when waxed.