Try thinking from the children's points of view. Kids want food, warmth, love, a home. They don't like chaos, violence, shouting, hunger, neglect. If you are providing a secure home you don't need to worry. If you are in circumstances that might result in a child being harmed, is it fair to the child for you to keep it?
If the safety of a child/baby is being compromised and all avenues to resolve it have been exhausted then SS have every right to take the baby/child away.
How many months are you and is your medical centre / GP aware that you are pregnant? You must go for regular checks.
how on earth did a 10 day baby get fracture and what measures have you taken to get him returned to you, if they still class you as an "unfit mother" then they may well have concern about your next child. May I ask how old are you ?