Hopefully your secretary would have some say in whether you 'did it' or not - and you don't just have her to boot. :)
Your wife has insecurity and esteem issues, which you can help her with by giving her the reassurance she needs and encouraging her to talk to a professional.
This is ultimately your wife's problem - provided you have done nothing to badly affect her confidence and self-esteem - and only she can deal with it.
When you get home, briefly discuss your working day and then change the subject. Refuse to be drawn, and tell her you spend all day in the office and you want to get away from it when you're at home.