Wednesday, March 25, 2009

It's a carefully designed science experiment, so why are people mad?

Good science experiments are hard to do, especially when it involves human subjects. That's why, as a scientist, I get upset when I see people knocking a well-designed experiment because they are mis-interpreting the conclusions.

The experiment I'm talking about is a study on weight loss published in the New England Journal of Medicine that found that the amount of weight lost by a person on a diet is determined solely by the difference between the number of calories a person consumes and the number of calories that person uses. It doesn't matter if the person is on an Atkins diet, or a Subway diet, or a nothing-but-chocolate diet. Simple enough, and, frankly, not surprising. But people are complaining. So, let's step through this. First, I'll go through the experiment and explain why I think it is scientific. Then I'll go through the complaints I've seen. And I'll end on why these people are wrong in attacking the study, even though the arguments they raise are valid. Let me start by saying I am going off of published summaries of the diet study; I haven't read it myself. If I screw up in my interpretation, then feel free to blame me for not reading the original study. (Since I'm not a medical doctor, I feel I'm likely to not understand or mis-read the professional journal article. I barely understand a large part of the astronomy literature.)

First, the study itself. There have been many studies on the efficacy of different diets, with many conflicting results. The reason for this is, in my opinion, is that many of these studies don't limit the variables. They find a pool of volunteer subjects, give them a diet to stick to, and then follow their progress. But people make lousy experimental subjects. We are all different, so genetic differences are always present. People lie about whether they stuck to their diets, because they feel ashamed when they cheat. The same food can have vastly different numbers of calories, depending on how it is prepared. People lose interest over time. These are very difficult variables to control

But these studies also often ask questions that render these variables crucial. For example, if I ask the question, "Does the Atkins diet or the Chocolate diet lead to greater weight loss after a two year period?", I'm asking for inconclusive results. What if the Atkins group gets tired of counting carbs after 6 months, but the chocolate group is happy to continue eating their allotment of Hershey bars? Maybe the Atkins meals take an hour to prepare, while the Chocolate meals are instant, meaning that the time-pressed subjects in the chocolate group are more apt to stick to the diet on a night when the kids have soccer and Girl Scouts, the spouse has a PTA meeting, and the subject has five hours of take-home preparation?

So the experiment started with a more pointed, direct hypothesis: The amount of weight lost on a diet is only due to the difference between the number of calories consumed and the number of calories used. They decided to test this hypothesis by varying the types of food consumed in a controlled manner, including diets that were low in fat, or low in carbohydrates, or low in protein, and a few other similar diets. The scientists used a large group of people and held them to pretty rigorous standards of eating only from a select menu of foods, of requiring them to write down all of their food and exercise, of routine measurements of their metabolism (rate of calorie burning during an activity), and of requiring the subjects to be as honest and as sticktoitive as possible. And the study used a large group of people and followed them for a long time.

The results were clear: the amount of weight lost by a person depended only on how many calories they ate and how many they used, not the type of diet. This is a nice, clean, clear result.

So, of course, the arguments start pouring in. Kathy Freston complains that the typical dieter will never follow the rigorous standards of the experiment. Another commenter complains that the study didn't consider the nutritive value of the diets.

But the study was not designed to test the question of whether the typical person would stick to the diet. It was designed to test the hypothesis I stated above. The very regimented meal plans and diaries were implemented to remove the variable of people's natural cravings to cheat from the equation. Nowhere does the study claim that people should follow their regimen. The conclusion form the study is solely related to the amount of weight lost as a function of consumed and used calories. It is indeed a very good question as to how to encourage people to stick to a diet, and whether certain foods might help someone stick to their diet by reducing hunger cravings. But this study was not testing that aspect, so it is unfair to attack the conclusions for not considering that variable.

Another complaint, on the nutritive value of the diets, is also irrelevant to the tested hypothesis. We know that humans need specific nutrients to remain healthy. And it may well be true that a certain balance of those nutrients will, by keeping a person healthy and/or full of energy, help a person stick to a diet or feel energetic enough to exercise. But, again, those variables were specifically excluded from the study. It is unfair to criticize the conclusions for ignoring a variable that was specifically excluded.

In short, the weight loss study was designed to test a specific question, and they came up with an answer to that question. The amount of weight a person loses in a diet depends only on the number of calories they consume and the number of calories they use. That conclusion should not be interpreted as saying that any diet is okay. The study says that I would lose weight if I ate nothing but 1200 calories of lard every day, but it was not designed to say that such a diet was healthy. I suspect that, if I went on the all-fat diet, I would rapidly become ill, malnourished, and unmotivated.

The results of this study, that a calorie is a calorie, can now be used in the interpretation of other diet studies. Dietary scientists no longer have to worry that our bodies view fiber and fat calories differently, because they don't. They can now design an experiment to test the other variables that this study purposefully limited, like the impact of nutritive balance, or the relative feeling of fullness that different foods provide, just what the critics of this study wanted to see.

In short, we shouldn't criticize individual scientific studies for not testing every possibility. This is because the best scientific studies purposefully limit the possibilities to ask and answer a specific question. And when a study does answer a specific question, we must be very careful to not draw any inferences beyond those supported by the experiment and its conclusions.

1 comment:

  1. I don't have a favourite diet to defend. My beef with such diet claims are the use of words like "solely". This wasn't shown. Not all possible permutations were tested. One might think that this is a minor quibble, but here's where it goes.

    If the claim is that weight loss is only dependent on calorie intake vs. calories burned, then this is clearly wrong, physics notwithstanding. Evidence can be had at any University with a cross country team.

    It's nearly impossible to make a dent in your calories in vs. out through exercise. The slogan is, "run a mile and eat a donut, and you're even." But what happens if you can make a serious dent?

    When i was on the cross country team, i was a poor starving student. If i couldn't eat it at the cafeteria, i couldn't eat it. While on the team, my intake didn't change from before or after. It certainly didn't go up, because i could barely get to the cafeteria as it closed. I couldn't go back for seconds.

    But my exercise level was off the normal scale. The workouts were 15 miles, twice a day. That's 30 miles a day, for 210 miles a week. These were 6 minute miles. If the 15 miles workout took longer than 1.5 hours, then i'd miss the cafeteria for that workout. That would be breakfast or dinner.

    Now, Dr. Cooper doesn't use calories to talk about exercise. He talks about ml's/kg/min - which is a rate, expressing power, not energy. Then he converts a rate and distance to a point system. And he suggests 30 points a week as a goal.

    But a 6 minute mile earns you 6 points, and so 15 of them in a row earns you 90 points. And 30 of them in a day earns you 180 points. For a total of 1260 per week. So we (on the team) were doing about 40 times more than a normal healthy individual.

    Dr. Cooper doesn't seem to give a conversion from points to calories (really kilocalories). But the "eat a donut" thing is talking about an 8 minute mile. That only gets you 4 points. So running an 8 minute mile six times a week gets you 6 donuts a week. Cross country training gets you 250 donuts a week.

    If there are absolute calories, that is (in - out) = weight loss or gain, then one would expect that eating fewer than 250 donuts a week would make a cross country runner drop pounds in a hurry. But i gained weight. About a pound a week. Now you could say it was all muscle mass, and you'd likely be right. But i had (and have) some tummy fat, and it went nowhere.

    Here's what works for me. If i reduce my intake by a small amount, i lose weight. Exercise makes me feel better, but is ultimately irrelevant to weight loss. That's me. YMMV.

    So here's another way to look at it. I know someone who gains weight if she consumes more than 500 calories a day. This is maybe 25% of my diet. So where do my other 1500 calories go? IMO, my body sheds them. But it doesn't have to. If i eat a lot less, my body will simply shed a lot less, and though i'll be hungrier, i won't drop any weight. Physics may come into the picture eventually, but it's very hard to see where.