There are two agreements (contracts) in existence here. One between you and the loan company, the other between you and your ex.
The first contract has nothing to do with your ex, and is simply an agreement between you and the loan company. If you default, it's you they will pursue and it would be your credit rating that would suffer - not your ex, and his/her credit rating.
The second agreement between you and your ex is still a contract. You were obliged to lend him money and he was/is obliged to repay it over a four year term. If he defaults you can raise an action for breach of contract at a sheriff/magistrates court - your remedy most likely being damages, or perhaps, implement/performance.
The bit of paper you have seems sufficiently clear to indicate that it is a loan agreement and not a statement that a gift has been made. Even better if all the terms and conditions and the word 'loan' are clearly stated - particularly an obligation to pay monthly or at least regularly. If the regularity of the payments is unclear he may try to argue he intends to pay you on an ad-hoc basis irregularly over the four years, but repay the entire balance by 2011.