Yep, Smowball, excuse the analogy but this is very much an emotional zit or boil and needs to be lanced soonest or it will turn into a major abcess. The "discussion" lancing will put you back on track (on your terms and list them mentally, as ummmm rightly points out) or you know where you stand if he refuses to properly discuss, throws a paddy or walks out.
It might be worth preparing yourself with your solicitor, so you can serve quickly and before he does (just a contingency.....)
Choose when you go into battle carefully and the mood of him - a bad day at work will not bring you both any favours - and I go back to having this discussion out of the house so that it is seen on neutral territory and that he (or you) can't afford to be too emotional.
You might want to draft and send a letter as this is often a way of communicating your seriousness. Obviously, don't write and immediately send it, give it overnight and re-read and re-write what you want; often we down-tone the emotional content and, in doing so, we make the letter more powerful (again focus on the strengths and + points before launching into the negs).....and then finish with an invitation to discuss the letter in his own time but by a certain date, not too far off so he can't hide behind it, thinking this will all go away.
Be brave too.
And this comes from a male!