As with, say, a planning application to build a house, the fee covers only consideration of the actual application by the local authority, not the actual planning permission (or sex shop licence) itself. i.e. it's non-refundable in the case of a refusal. (To apply for planning permission to build a house can cost a lot of money; you still have to pay if planning permission is refused though. It's the same with licence applications made under the Local Government (Miscellaneous Provisions) Act 982 [as amended]).
Schedule 3 of the 1982 Act requires local authorities to charge a 'reasonable fee' for considering licensing applications but does not elaborate upon what such a fee should be or as to how it should be determined. So, as long as they comply with the 'reasonableness' provision, each local authority is free to set their own fee.
Cheshire East Council, for example, sees £2,280 as a 'reasonable fee' for a first application for a sex shop licence.
Therefore you have NO chance of getting your money back simply because your application was refused. (As I've stated above, the fee is ALWAYS non-refundable).
The only chance you might have of getting at least SOME of it back would be to seek a ruling from the Local Government Ombudsman, declaring that the fee you were required to pay exceeded the definition of 'reasonable' within the legislation.
http://www.lgo.org.uk/