One site suggests the following: "...as underworld slang for a prison guard dating back to the mid-19th century, 'screw' was suggested by someone harsh and brutal, one who used thumbscrews on prisoners." From the "Encyclopedia of Word and Phrase Origins" by Robert Hendrickson (Facts on File, New York, 1997).
Screw as a term for a prison guard is based on the fact that screw was originally slang for "key." One of the most important functions of a prison guard, or turnkey, as he's often called, is to see that prisoners are locked up at the appropriate times -- and that involves turning the "screw." Interestingly enough, Henry Mencken reports in The American Language that in the 1920s deskmen and bellboys in hotels used screw as a slang term for room key. Another theory is that screw refers to the thumbscrews used by jailers in ancient times to torture prisoners into confessing.
From Morris Dictionary of Word and Phrase Origins, Second Edition (1988) by William and Mary Morris
Eric Partridge, in A Dictionary of Slang and Unconventional English, Fifth Edition (1961), also wrote:
*screw. A skelton key: c[ant]: 1975, Potter; slightly ob[solescent]. --2. ? hence, a turnkey or prison warder: 1821, Egan: c[ant] until ca. 1860, then low s[lang]....