I would say, cut the ties, distance yourselves. Families do do that, for whatever reason. Life is too short to spend time and be influenced by those who make you unhappy. As long as your husband agrees, and has so guilt about distancing himself from his father, go for it. It's a hard one but I have cut free from friends who dragged me down, in recent years. Family is more difficult as there should be loyalities, but if it's not working, don't feel bad about cutting loose. Good luck to you.