Depends what you mean by castle, and whether you want it to still be standing.
There must be lots of stone hill-forts which could qualify on a looser definition. There's one I've seen in Wales, at Carn Goch ("red rock") near Dynefor (Dunevor), and if I remember right it's some 4,500 years old (Bronze Age?).
We Brits were at it long before those Norman upstarts...