By any sensible measure Michael Phelps ought to be at least in the top two. Most Medals overall, most gold medals, most individual medals overall, most individual gold medals, and dominant over four Olympic Games. Also these have come at several different swimming disciplines across the distances from 100m to 400m.
Bolt's victories are perhaps more memorable as I think there's a general bias in favour of athletics. I don't think I'll ever forget his wonderful run at the 2008 100m final, when he stopped racing after about 85 metres and still broke the world record with clear air between him and the rest of the field.