Good science experiments are hard to do, especially when it involves human subjects. That's why, as a scientist, I get upset when I see people knocking a well-designed experiment because they are mis-interpreting the conclusions.
The experiment I'm talking about is a study on weight loss published in the New England Journal of Medicine that found that the amount of weight lost by a person on a diet is determined solely by the difference between the number of calories a person consumes and the number of calories that person uses. It doesn't matter if the person is on an Atkins diet, or a Subway diet, or a nothing-but-chocolate diet. Simple enough, and, frankly, not surprising. But people are complaining. So, let's step through this. First, I'll go through the experiment and explain why I think it is scientific. Then I'll go through the complaints I've seen. And I'll end on why these people are wrong in attacking the study, even though the arguments they raise are valid. Let me start by saying I am going off of published summaries of the diet study; I haven't read it myself. If I screw up in my interpretation, then feel free to blame me for not reading the original study. (Since I'm not a medical doctor, I feel I'm likely to not understand or mis-read the professional journal article. I barely understand a large part of the astronomy literature.)
First, the study itself. There have been many studies on the efficacy of different diets, with many conflicting results. The reason for this is, in my opinion, is that many of these studies don't limit the variables. They find a pool of volunteer subjects, give them a diet to stick to, and then follow their progress. But people make lousy experimental subjects. We are all different, so genetic differences are always present. People lie about whether they stuck to their diets, because they feel ashamed when they cheat. The same food can have vastly different numbers of calories, depending on how it is prepared. People lose interest over time. These are very difficult variables to control
But these studies also often ask questions that render these variables crucial. For example, if I ask the question, "Does the Atkins diet or the Chocolate diet lead to greater weight loss after a two year period?", I'm asking for inconclusive results. What if the Atkins group gets tired of counting carbs after 6 months, but the chocolate group is happy to continue eating their allotment of Hershey bars? Maybe the Atkins meals take an hour to prepare, while the Chocolate meals are instant, meaning that the time-pressed subjects in the chocolate group are more apt to stick to the diet on a night when the kids have soccer and Girl Scouts, the spouse has a PTA meeting, and the subject has five hours of take-home preparation?
So the experiment started with a more pointed, direct hypothesis: The amount of weight lost on a diet is only due to the difference between the number of calories consumed and the number of calories used. They decided to test this hypothesis by varying the types of food consumed in a controlled manner, including diets that were low in fat, or low in carbohydrates, or low in protein, and a few other similar diets. The scientists used a large group of people and held them to pretty rigorous standards of eating only from a select menu of foods, of requiring them to write down all of their food and exercise, of routine measurements of their metabolism (rate of calorie burning during an activity), and of requiring the subjects to be as honest and as sticktoitive as possible. And the study used a large group of people and followed them for a long time.
The results were clear: the amount of weight lost by a person depended only on how many calories they ate and how many they used, not the type of diet. This is a nice, clean, clear result.
So, of course, the arguments start pouring in. Kathy Freston complains that the typical dieter will never follow the rigorous standards of the experiment. Another commenter complains that the study didn't consider the nutritive value of the diets.
But the study was not designed to test the question of whether the typical person would stick to the diet. It was designed to test the hypothesis I stated above. The very regimented meal plans and diaries were implemented to remove the variable of people's natural cravings to cheat from the equation. Nowhere does the study claim that people should follow their regimen. The conclusion form the study is solely related to the amount of weight lost as a function of consumed and used calories. It is indeed a very good question as to how to encourage people to stick to a diet, and whether certain foods might help someone stick to their diet by reducing hunger cravings. But this study was not testing that aspect, so it is unfair to attack the conclusions for not considering that variable.
Another complaint, on the nutritive value of the diets, is also irrelevant to the tested hypothesis. We know that humans need specific nutrients to remain healthy. And it may well be true that a certain balance of those nutrients will, by keeping a person healthy and/or full of energy, help a person stick to a diet or feel energetic enough to exercise. But, again, those variables were specifically excluded from the study. It is unfair to criticize the conclusions for ignoring a variable that was specifically excluded.
In short, the weight loss study was designed to test a specific question, and they came up with an answer to that question. The amount of weight a person loses in a diet depends only on the number of calories they consume and the number of calories they use. That conclusion should not be interpreted as saying that any diet is okay. The study says that I would lose weight if I ate nothing but 1200 calories of lard every day, but it was not designed to say that such a diet was healthy. I suspect that, if I went on the all-fat diet, I would rapidly become ill, malnourished, and unmotivated.
The results of this study, that a calorie is a calorie, can now be used in the interpretation of other diet studies. Dietary scientists no longer have to worry that our bodies view fiber and fat calories differently, because they don't. They can now design an experiment to test the other variables that this study purposefully limited, like the impact of nutritive balance, or the relative feeling of fullness that different foods provide, just what the critics of this study wanted to see.
In short, we shouldn't criticize individual scientific studies for not testing every possibility. This is because the best scientific studies purposefully limit the possibilities to ask and answer a specific question. And when a study does answer a specific question, we must be very careful to not draw any inferences beyond those supported by the experiment and its conclusions.