I would imagine this would only be applicable on late night trains, when harassment mostly occurs.
Sorry but this would definitely need to be policed. I have got the last train home from the City on many an occasion (affectionately known as the Vomit Comet, for probably obvious reasons) and 99% of passengers are drunk. You would never get women only carriages policed by comment consent when practically every bloke is wasted.
As someone who has had to put up with plenty of hassle from groups of rowdy City boys (ok, when I was a bit younger) I would use a women-only carriage. It's all very well saying men's attitudes should change instead, but that aint happening very soon!