I have long been interested in the facts around calorie determination because much of what we do in nutritional sciences obviously revolves around this core concept. Unfortunately I believe calorie counts are flawed, fundamentally so, and this video snippet from Scientific America helps explain that point a little.
A key point to take from this all is that if we want to lower net calorie (energy) intake for weight loss goals choosing unrefined, whole and fibrous foods will help because we generally spend more energy digesting them. Conversely for weight gain we need to consider the form of food too. Cooked vs raw food will also impact the calorie availability and as Ferris Jabr explains due to other physiological issues not every person will extract the same calories from the same food prepared the same way.
It is all a bit hit and miss really to follow calorie counts strictly. However we are creatures of habit and we can work out where we might be going wrong for whatever calorie related goal we have and adjust our normal intake with these elements in mind.