my bf hurt his little finger while playing football. Well actually, it was after he made a mistake which led to a goal and he punched a wall... Anyway it swelled up down the side of his hand, but he didn't go to the hospital for a week. Then he was told it was broken but there was nothing they could do about it by now. He was just told not to play golf etc for a few weeks, but no splint or anything as the bones had already knitted together (in the wrong way). Now it is permanently bent, he can't straighten it properly, and his knuckle is misshapen. He's really hacked off that he didn't go to A&E straightaway. At least it is only his little finger - doesn't make much difference to his life. Three fingers would though, might affect your grip and strength. They told my bf that they don't bother re-breaking fingers and re-setting them, because the bones are so small and lots could go wrong, leaving them even worse than before.
Really, you could be putting yourself through a lot of unnecessary hassle by not going to the doc's. Make sure you go as soon as you get back.