The Lord or Squire would probably either not need to give his workers time off or, alternatively, have a vested interest in doing so.
Firstly, cricket was often played by 'gentlemen', who didn't need to work for a living. Within living memory for some, first class teams had both 'players' (= professionals) and 'gentlemen' (= wealthy amateurs) among their number.
Secondly, cricket was often at the centre of substantial gambling. The Lord or Squire would be happy to let a worker have a few days off if he could score a century and take a few wickets, thus helping to win his boss a great deal of money ;-)
It's interesting that you you should mention Hambledon Club. The Wikipedia entry for that club makes reference to a two-day match (between Kent & 'Hampshire & Sussex') being played in 1772:
http://en.wikipedia.org/wiki/Hambledon_Club
Chris