Old bather, ignore calories, they're unimportant in the grand scheme of things.
When you remove fat (that much maligned nutrient) you also take with it the fat-soluble vitamins....in this case, Vitamin D. Vitamin D also facilitates the absorption of calcium, ergo you remove the natural fat content, you remove vital nutrients.