Agree, Naomi, also, when Abraham reconstructed the Ka'aba, he was making use of an existing building which (as I recall) used to house idols from previous religious beliefs. The Ka'aba is hollow, it is opened annually for cleaning (it's a major ceremony, I have a friend who was there one year on opening day and got to see inside).
Certainly the Ka'aba itself is not worshipped, any more than the Black Stone, which is incredibly old and could well be a meteorite. The comparison to churches facing to the Holy Land is good since Muslims the world over face Mecca to pray, it being the No. 1 holy city in Islam. Anyone suggesting that the Ka'aba itself is being worshipped is wrong, the Ka'aba is situated in the centre of the Grand Mosque (Haram Sharif) in Mecca, Muslims would no more worship a building than Christians would worship (say) Canterbury Cathedral. It's a precious holy object in a place of worship, not an object TO worship.