Thursday, March 23, 2006

Talk About Creativity!

Usually when researchers find no measurable differences between groups they're observing, they'll either choose to tell it like it is and reach the honest conclusion that there was no difference, or they'll simply not publish their findings.

This month however, it appears researchers got creative to find statistically significant differences in the publication of Weight gain over 5 years in 21,966 meat-eating, fish-eating, vegetarian, and vegan men and women in EPIC-Oxford, published in the International Journal of Obesity.

I say "creative" because the research team abandon the traditional measurements of pounds and/or kilograms and instead reported their findings in grams! Not only did this sleight-of-hand help massage the data to statistically significance, it also got the attention of the media as we see in the Globe and Mail article, Need a carrot to stick to vegetarian eats?

Creative statistics may get you in the media, but it's intellectually dishonest and purposely misleading. In two words, bad science.

So, what's the hub-bub about?

The researchers purport to show there is a statistically significant difference between different dietary habits on weight gain over a period of five years. The dietary patterns observed included meat-eaters, fish-eaters, vegetarians and vegans. At the end of the five years, there was no difference between the groups. This is stated, very clearly, in the abstract: The differences between meat-eaters, fish-eaters, vegetarians and vegans in age-adjusted mean BMI at follow-up were similar to those seen at baseline.

But then we find creativity at work with the data - after massaging the data to death, it's finally found that vegans gained less weight annually than fish-eaters; who in turn gained less weight than meat eaters. The conclusion - During 5 years follow-up, the mean annual weight gain in a health-conscious cohort in the UK was approximately 400g. Small differences in weight gain were observed between meat-eaters, fish-eaters, vegetarians and vegans. Lowest weight gain was seen among those who, during follow-up, had changed to a diet containing fewer animal food.

How much weight are we talking here? Oh, an ounce or so. So completely insignificant and totally able to be skewed by something as benign as a participant drinking a glass of water before they weighed.

But, convert the ounces to grams and, viola!, you have differences that suddenly reach statistical significance deemed worthy of publication, press releases and media attention!

What we really have here is data that's completely worthless - it offers us nothing more than a lesson in data massage and creative presentation. At the end of five years there was no statistical difference between the groups in pounds or kilograms, and there certainly was no clinical significance to support the media attention or recommendations making the rounds to eat less animal foods.

1 comment:

  1. The biggest problem I have with the article is not the use of the metric system, which is after the more scientific measurement system, but rather that the weight gains should be expressed in kilograms rather than grams.

    To reexpress part of the abstract

    The mean annual weight gain was 0.389 kg (SD 0.884 kg) in men and 0.398 kg (SD 0.892 kg)in women. The differences between meat-eaters, fish-eaters, vegetarians and vegans in age-adjusted mean BMI at follow-up were similar to those seen at baseline. Multivariable-adjusted mean weight gain was somewhat smaller in vegans (0.284 kg in men and 0.303 kg in women,...

    The differences do not seem so great. Also note that the standard deviation of weight gain is actually greater than any difference between meat eaters and vegetarians.

    Thanks for the space to comment.