My opinion:
Yes, you should invite your colleague. Other people at work will know that you and her are good friends and, given that you're only having a low-key wedding, see nothing unreasonable in you only inviting her & her hubby.
With regard to your brother, I'd phone him and say "We've not always seen eye-to-eye in the past but I'd still be happy to see you at our wedding. However I won't be offended if you'd rather not come". That way you've invited him (to satisfy your mother) but hinted that you're not desparate for him to attend. The chances are that he'll decline anyway (if only to save on the cost of a wedding present!)