"In relation to Earth, that galaxy has not necessarily moved the same direction as us."
The motion is all relative, so you may take any reference point you wish. Where you sit at the moment, will be just fine.
There will be what for want of a better term I would call "genuine motion", as witnessed by the fact that not all galaxies are heading away from us: for example Andromeda seems to be heading our way. But as I understand it, most of the motion we see is down to this space creation causing inflation; so yes, generally speaking they are all heading away from us (or from any other point for that matter). I believe that, off at weird angles are not sufficiently significant / affective to be worth considering.