Poor little thing, hope he settles soon. It's difficult with rescue pets, you may never know but if behaviourable then patience, love, consistency and time can overcome a lot. I've always had rescue cats and it is a lovely feeling when you see how far they have come, even if it is only little things but a big thing for them.
One of mine won't go near anyone but me though to me she is the most loving cat, incredibly affectionate - anyone else comes near her she will run. After little bits of gradual light non-threatening exposure to other people over the two and a bit years since I rescued them she let my friend give her a little stroke the other week - a huge thing for her!
The other, her sister, has taken a long time to relax affection wise and she has come a long long way - she's now a near permanent fixture curled up on my sofa/bed by my feet and has even started to try to mimick some of her sister's affectionate ways and is a much happier and affectionate cat.
With rescue cats I don't expect more than I get from them so every step they make is something special.