Try going into Task Manager to take a look at what (your PC thinks) might be using it. I've found that some programs (e.g. Firefox) can carry on running for several minutes after they appear to have been closed. Any program that has been used to send data to your external drive, or to retrieve data from it, during the current session could be to blame. (I've also had problems ejecting external USB drives but I've always managed to locate the rogue program, usually Firefox, via Task Manager).
It's also worth remembering that it's always perfectly safe to remove a USB device after the computer has been shut down.