You need to exfoliate the dead skin off as the hairs are thinner they cannot break through the dead skin as easy, sloughing this off with body scrub once to twice a week depending on how dry your skin is and also moisturising your skin after each bath or shower should prevent ingrown hairs as much, you might still get the odd one but this will help.
Yes get them out if you can. I didn't and had to pay for private hospital treament for a pylonial sinus removal (ingrowing hair). It was incredibly painful and took at least 2 months to get better, Now if I get one in bikini line I scratch it open and pluck hair out. Horrible but far better in the long run.