Sounds like your manager is doing what managers tend to do, rely on goodwill to cover up their lack of talent and management skills.
Ignore any chat about hours, you have your contract and terms that you're happy with, you've helped out in the past, now it's up to those who get the big bucks to deal with it.