I used to work in the fitness game - some of that time spent as a life guard. I can still think of about 6 peple who would swim an average of 5 times week, non-stop, at a great pace, for up to an hour, but did not appear to lose an inch. I was dead fit at the time and had little body fat, but couldn't swim a 5th of their distances.
Me and a couple of the "big" swimmers, who could also lift weights that would put most to shame used to discuss this, and all that we could come up with is that swimming doesn't tend to raise the heart rate as much as weight bearing exercise (walking , running etc). I'm not sure if this could be down to temperature, different breathing methods or the supportive factor of the water.
Anyway, this is anecdotal, not scientific evidence. I'd recommend getting a heart rate monitor (cheaper and easy to use nowadays) and working out your target rates e.g. for weight loss /cardio training etc. This is a good motivator.
You can do sit ups, crunches etc all day, but if you don't do the cardio stuff (i.e. elevate the heart rate safely and consistently) you're not going to see the results of all that hard work.
All the best.