If it's a 'proper' direct debit then you're protected by the Direct Debit Guarantee, which means that you can stop it at any time (by simply notifying your bank) and your bank must refund any payments taken from your account without your authorisation.
However many people provide ISPs (and others) with their debit card details and refer to that as a 'direct debit', but it's not. It's a 'continuing authority' which you CAN'T cancel. Only the recipient can cancel the arrangement. If the recipient takes more than they're entitled to, it's entirely a matter between the account holder and the recipient. The bank can't intervene.
Chris