Memory: Most cameras require you to buy memory cards to store your pictures on. These add to your costs. Ideally you need a camera which has a small amount of built-in memory (which can hold a few pictures at a time) but allows you to use a memory card (for storing more pictures) at a later date, if you wish to do so.
Batteries: Some cameras use built-in rechargeable batteries, while others use 'ordinary' batteries (e.g. 'pen cells'. There are advantages and disadvantages to both systems but I prefer cameras which use 'ordinary' batteries. You can always buy somespares at a garage or corner shop if you run out of power. (Although I've referred to 'ordinary' batteries, I strongly recommend buying the lithium variety. They usually last far longer than normal alkaline batteries).
Vivitar is a well-respected brand name at the cheaper end of the market. Their 3 megapixel V3188 model uses AAA battteries. It's got an 8x digital zoom, built-in flash and a macro mode. It's also got enough internal memory to store your photos from a single angling expedition, but it can also accept an SD memory card to hold more pictures. It's lightweight and will slip easily into a pocket of your tackle bag:
http://tinyurl.com/V3188-Argos
Chris