As has been stated, it seems a case of just your word against his, and as this man does not sound the type of person the courts would class as an upstanding citizen they are unlikely to take his word against yours, so I would doubt very much this would result in you 'losing everything'. As you say, you have a clean record and a 'good job', whereas the courts are not unbiased and the fact this man does not work and also is known to use drugs will not work in his favour, especially if he has a criminal record.
I am not saying I agree with that type of prejudice, it is my view they should deal with the case as it is and the evidence they have of it, whether or not he works or what he does in his spare time in my view should be immaterial..... but that is not how the courts think. ....but from what you say he sounds an unpleasant character so he does not gain my sympathy either. Sounds like he has a touch of drug induced psychosis and could do with some help!
Have you told them about his endless provocations and admitted to pushing him on that one occasion? If you do so it may work in your favour, as they may be then be even less likely to doubt your word.