Presumably in the 1st line of your question you mean 'buyer' not 'seller' - you're selling, he's buying, right?
If that is so and you have had nothing more than an informal conversation with an estate agent to discuss acting on your behalf, then you are not contractually bound. But then the intriguing thing is how the prospective buyer has made contact with you. These agents are quite canny at this, and aren't renown for disclosing the identity of a prospective buyer that they may have identified until after you have signed up to a contract.
If one signs up to an agreement with an agent it is essential to disclose any prospective buyers whose purchase of the house is excluded from the agency agreement - this is most important when one changes agent and there is a prospective buyer already in the pipeline who is not proceeding, but who might come back later. Otherwise it is possible that one ends up with a legal obligation to pay both agents.