There are two aspects to this.
There are a few driving offences where no “endorsement” on the driver’s licence is made. However, most offences do carry an endorsement. This has nothing to do with penalty points or disqualification, but is simply a record of the offence which is provided on the driver’s licence and on his driving record held by the DVLA. The endorsement usually lasts for four years from the date of conviction although, as chas correctly points out, some offences (those involving drink or drugs) carry an endorsement which stays on the licence for eleven years. After four (or eleven) years the driver can apply to the DVLA for a new licence with the endorsement removed. For these purposes the date of the offence is usually the starting point of the endorsement period, though for a few offences it is the date of conviction which starts the clock.
As well as this, as part of the sentence, (unless there are “special reasons”) the court must usually either disqualify the driver or award between three and eleven penalty points. (Note, not both. No points are imposed if a disqualification is ordered). There is a “totting up” law which says that if a driver accumulates twelve or more penalty points within three years then he becomes liable to a mandatory six month disqualification (unless he can argue that “exceptional hardship” will follow if he is banned). (The period is ten years in the case of the drink/drugs offences. Similarly the dates of the offences being normally used as the starting point, though for some offences it is the dates of convictions).
So the answer to your question is really that, in the fourth (or eleventh) year, although the endorsement is still shown on your licence the penalty points are no longer valid for totting up purposes.
Hope this helps.