Does this work?
Tools -> Preferences
Show All Settings -> Click the plus next to Interface -> Main Interface
Change the Interface Module to Qt Interface
Click the plus next to Main Interface -> Qt
Change the Window Opacity to the desired amount
Close and reopen VLC for the change to take effect
taken from
http://superuser.com/questions/378770/vlc-always-on-top-and-transparent-when-out-of-focus
I haven't had a transparency/opacity issue but decided to give this a go and it lets me vary the opacity.