We have had cats for many years,and have two at present,so I may be able to give you some (personal)advice.
You say that you have "just " rescued Ginger.
Cats are "place" animals and feel secure in surroundings that they know.
I think it would be very inadvisable to take Ginger with you,please find a way to leave him at home if at all possible.
He would probably feel that you were taking him to be rehomed again,given that he hasn;t been with you long, and on your return could take a lon while to settle again as he would be anxious.
He seems to get on with your two.
Taking him with you poses the risk that he will get out,and get lost,not something you would want to face.
I don't think that the feeding "problem" cannot be overcome.I assume that you have someone coming in to feed the other two?
Just arrange with them to stay while the cats eat their food,and feed Ginger in another room with the door closed.Don't let Ginger out until the other two have finished.If the person doing this cannot agree to this they shouldn't be looking after the cats.
Leaving him at home would be far less traumatic for Ginger,and he wouldn't have to re~intergrate with the other two on his return.
If you had had Ginger for some years there wouldn't be any problem taking him away,but as he is relatively new it would be kinder to leave him in the place he regards as his ow,with his "friends".
If this doesn't sound right,let me know,and I will see if I can think of something else.