Unless you might mean the Scott Monument as some regard the castle being "on the other side" because the railway passes between the Gardens and the Castle. The Gardens are directly below the monument.
Sorry, KARL, one part of the Gardens is between Princes Street and the railway line, and the other part is between the railway line and the foot of Edinburgh Castle Rock.