The number of people who buy it...pure and simple. Bookstores keep a record of which books are selling in quantity and the figures are added up throughout the country/world. Thus, a list of the Top Ten, say, can be produced. At the moment, the latest Harry Potter is slugging it out with 'The Atkins Diet' for top position.
I guess the term is pretty meaningless unless it is also qualified in some way e.g. 'Current 10 bestsellers', or 'Top 100 Bestsellers of 1991'. Just to say a book is a 'bestseller' doesn't really tell you anything unless you know the criteria by which it become regarded as such. I suspect the term has been hijacked by marketing people and really just denotes that the book is expected to sell 'a lot' of copies.