Your computer's BIOS is the collection of settings which tells the PC how to operate even before Windows has loaded.
When you first start your PC, you'll see something on your screen along the lines of 'To enter Setup, press Delete'. (As Billy has indicated, different PCs use different keys).
If you hit the relevant key, you will see a screen which allows you to review and change the BIOS settings.
Unfortunately, the exact layout of the screen varies between different PCs but it's all fairly straightforward. Because your mouse driver won't have loaded, all navigation within the BIOS is done by using the arrow keys, some of the F keys (e.g. F7), the 'plus' and 'minus' keys, 'Enter' and 'Esc' . (It's not as difficult as it sounds. Instructions appear at the bottom of the screen).
Within the BIOS settings are the instructions which your PC uses when seeking a drive to boot from. (These are typically, first=floppy, second=hard drive, third=CD). It's possible that your BIOS requires the PC to check whether there's a bootable drive attached to a USB port. If so, all you have to do is to change the order so that the hard drive is checked before the USB drive.
If the USB drive is not specifically listed as a possible boot source, then you'll need to check further through the BIOS settings to see if there's a line like 'Alllow boot from USB drive?'. If there is, you'll have to change 'enabled' to 'disabled'.
I know that can all sound rather daunting if you've never been into the BIOS before but it's actually very easy. As long as you're sensible, you won't 'muck up' your PC. It's perfectly OK to press a few keys just to see what happens. Nothing will actually be changed unless you hit the key (typically F7) which the on-screen instructions designate as the 'Save' key.
Chris