Because of the supposed stupidity of the bird of that name, the word 'goose' has meant a simpleton since the 1500s. The word 'boo' in this context is the sudden, loud version of it...the one adults sometimes use to surprise or 'frighten' children...rather than the disapproving anti-cheering version. So, 'saying boo to a goose' - whether human or bird - is about the least brave thing one could imagine doing and, as a result, very shy people are accused of being unable to do even that.
We might actually really be afraid to say 'Boo!' to a ghost, on the other hand.