First of all, you absolutely DO have a say in these proceedings.
Taking more children to live with you is a massive step, and not something you signed up for when you got together with your partner.
You should feel comfortable in discussing your feelings with him, and if you donp;t, then this is a sign that this will nbot be a good move for anyone involved.
I understand your urge to give the children a secure and safe home, what mum wouldn;t want to do that - but you have to step back and remember that they are not your reponsibility, and it sounds as though your partner's ex would be delighted if they could become so.
Raising children that are not bologically your own is tough - I've done it, so I know - and you have to be absolutely sure that you and your partner are a strong unit, or the children will simply play you off against each other.
I suggeset you and your partner try some sessions with Relate, which wil enable you to clear away some of the emotional fog that is clouding the issues at the moment.
Even if you sort it out withour professional help, you must be very very sure it is what you want, and not just go along becuase you feel it is the right thing to do.
No-one is going to stand there with a gold medal because you 'did the right thing'.
This is real life - yours and your baby's, as well as your partner and children. You must be happy with what you decide - and if you have doubts -as yo do, hence posting, then you need to resolve them first.
Good luck - keep us advised.