Just as 'Cor blimey!' was originally 'God blind me!', so 'Darn your eyeballs!' was originally 'Damn your eyeballs!' In both cases, someone's eyesight - a vital faculty - was put under a form of divine threat.
yes, if grannies lost their knitting needles they often used to do their darning with bread knives instead. This is hard enough with woollen clothing, but very tricky indeed when you're trying to darn eyeballs; accidents have been reported from time to time.