No, it is in fact extremely good for you. See here : http://www.healingdaily.com/detoxification-die t/olive-oil.htm
Mothers in various countries also massage their babies with virgin olive oil. Like everything, moderation is the thing to consider and I think you are doing the right thing. Countries where virgin olive oil is used a lot in diet and skin care on the whole have far less incidences of some cancers (colon cancer), heart disease etc.