I agree with lankeela, grange - it is so much more kind to all concerned if your son would let you go to his house to feed the cat when they are away. It's less disruptive for the cat, the cat will stay in his own surroundings and keep to his normal routine, and he'll be pleased to see you. We leave our three in the charge of a neighbour when we go away - it means she can keep an eye on the mail and stuff like that, too. Would this be possible for you for next time?
In the meantime, please don't let him out this time, even though the weather is ideal for a cat - if he lives nearby then your garden is probably on his normal patch anyway, he'll be off like a shot and go back home, cats don't understand fences, and if he goes missing (as he may well do, if the weather's mild and he enjoys outside), you will feel awful.