if you use ALT+TAB you can switch between (among) windows on the taskbar.
It might be the case that you are opening the new websites in the same window as the old one and so there will be only one window on the taskbar.
CTRL+N in Internet Explorer and Netscape Navigator opens a new browser window or you can do it through the file menu.
I don't know of any options that would mean that only one window would ever appear on the taskbar, so it is unlikely that you have clicked/pressed anything there.
Post a response if you would like any further explanation.