Yes, I think you might be over reacting.
He, must of met someone and then realised the time. cut him some slack. would you feel this way over a friend who needs to find a date. You were going home anyway.
I'd not mention it until he arranges to meet you again, then say
Well sure but if I get distracted by some other fit guy, I might let you get wet for half an hour first before I let you know I'm not coming.
by not saying anything now makes you understanding.
Saying something later means your not a push over.