Depends what type of lights.
Some will have an AA battery somewhere, others may have a soldered on battery.
If you mean the lights with separate solar panel, then the battery will be in the base of the light part. You can open them up, but you might find its a battery like in older mobile phones, or could be a battery pack that looks like 3 AA batteries. If you open that up you will find the batteries are connected together - so you could replace then, remembering to connect them together in the same way.