Pollan and Bittman, the Morano and Milloy of GMO anti-science

Why do food writers think they are competent to evaluate the scientific literature? I know of at least two who, based on their tweets, clearly are not. One is Mark Bittman, who we have previously chastised, and now also Michael Pollan who has been a bit more coy about promoting anti-science related to GMO. Now they’ve both been broadcasting the flimsy results of this paper – A long-term toxicology study on pigs fed a combined genetically modified (GM) soy and GM maize diet – published in the “Journal of Organic Systems”. Why do I feel like I’m reading headlines from Climate Depot or Milloy’s Junkscience? Because it’s the exact same behavior.

For all you budding science journalists out there, here is your first red flag, novel groundbreaking research is rarely reported in a such journals. Not to demean the smaller journals, good science is done there, but the quality of the publications must be one of the first factors taken into account when evaluating the significance of results published in the lay press. Note Reuters and Huffpo both published fluffy repetitions of “press release” evaluations of the study. Neither appears to show any skepticism or depth into the significance of the results, other results within the paper, or whether the fundamental conclusions of the authors are even supported by the data. Let’s do this now.

First, let’s describe the study. It’s a long-term (22.7 week) feeding study in pigs, with two groups of 84 pigs randomly selected to either receive GMO feed or non-GMO feed. During the trial all conditions are controlled, the feeds are found to be nutritionally identical (interesting given how GMOs has no nutrients!11!!!), and were obtained according to standard practices of pig farmers from similar local sources. The pigs were raised to the standard age they are when they go to slaughter, and were then killed and their bodies autopsied. While living the animals were evaluated by weights weekly, level of activity of pigs, level of contentment, skin problems, respiratory problems, eye problems, stool quality, blood biochemical analyses right before slaughter, and mortality. At autopsy organs were weighed and evaluated by veterinarians for evidence of tissue pathology.

Second, the findings. A good science journalist determines these by looking at the data, not by repeating whatever the authors tell them. Looking at the data there were no differences in any of the major variables evaluated by the study, such as weights, veterinary costs, illnesses, or mortality. No significant differences in blood biochemistry were found. At autopsy most organ weights were similar between groups. There was a statistically significant (but likely clinically-meaningless) increase (0.1kg vs 0.12kg) in uterus weights in the GM group. At pathology there were nonsignificant decreases in cardiac and liver abnormalities in the GM group (half as many), in stomach pathology there was one significant finding of more “severe inflammation” (on a 4-point scale from no inflammation to severe) in the GM group. This is the finding that has been amplified as variably “damning” or “concerning” depending on which source is reporting these dramatic new findings.

But since we’re skeptics here (real skeptics not like global warming “skeptics” in scare quotes) we ask, is it really?

Lets take a closer look at the data in table 3. Here are the relevant numbers:
Carmentable3data

While it is clear that along the severe inflammation row there is a difference, look at the moderate inflammation row immediately above it, and see if it changes your mind. What if we were to combine this table into a binary, no to mild inflammation vs moderate to severe? The numbers become GMO 41, non GM 38. Why would I look at it this way? Because pathologic scales of things like inflammation are subjective. (***Update It has been pointed out that the authors also didn’t actually do tissue pathology, instead they just graded how red the stomachs were on gross pathology, which also makes this assay totally meaningless. See full update below***) One should be very cautious about results presented on such a scale representing true differences especially given the next nearest population on the scale is reversed and eliminates your effect when the two groups are combined. Trying to make this objective data to suggest an association is very much trying to cram a square peg through a round hole, and would not fly on most reviewers’ reads of this data, and if I had been a reviewer I would have squashed this on this point alone. The fixation on one single data point in this table to the exclusion of the others and building the conclusions around it is unscientific. One needs to be a lot more cautious given the design of this study. Let me explain.

This is not hypothesis-driven work. They authors did not at the outset say, “we propose stomach inflammation will be greater in GM fed pigs because of x”. No. What they did was feed pigs two different diets and then go fishing for abnormal values. This is not necessarily wrong behavior, scientists go on fishing trips all the time looking to find significant effects. What is wrong is then publishing the results of your fishing trip! This is unscientific.

If you were to study some 20 variables in your study (these authors studied far more variables and I would actually expect more abnormal results then we have), and have a cutoff for significance at the standard arbitrary value of p = 0.05, one would expect, just by chance, that 1 of those variables will be significant. A good scientist then says, “well that’s interesting, let’s see if it’s real”, and then follows this study with a hypothesis-driven study specifically designed to study the apparent effect. When the single effect is then studied in isolation, with appropriate power, one should see if the result you found, perhaps by chance, is a real effect or not.

So what we have in this study is the first half of a valid study (the fishing trip) but no real hypothesis driven research to confirm if this 1 in 20 result is real. There is no molecular data to suggest a mechanism. They don’t further determine if it was the soy component or corn component on the diet. There are no follow up evaluations examining this effect alone, or trying to link ingestion of cry proteins on stomach inflammation. So far, one can only conclude that it’s just as likely that this result occurred by chance as it is to be an actual effect of feeding the pigs GM corn and soy. Now, is that “damning” or “concerning”? Concerning is even a stretch.

Third, it’s important for the good science journalist to interpret these new findings in the context of the literature, and perhaps consult an expert in the field to determine the significance of these results in context of the total knowledge in the field.

One should mention the extensive literature on the safety of GM foods. Other writers including Mark Lynas have evaluated this paper as well with similar conclusions as mine. Additionally, Mark points out the paper’s favorable interpretation of Seralini’s work – a bad sign. The authors appear to have ties to anti-GMO advocacy groups, and even thank Jeffrey Smith (the hysterical anti-GMO fake expert with no scientific or medical training). Andrew Kniss points out that he can’t replicate their result with the appropriate statistical test. I admit, I am confused about exactly how they calculated the p value, as in their methods they describe using t tests, Mann-Whitney and Chi Squared variably based on the distribution or categorical nature of the variables, so half the time reading I was trying to figure out which test they were using at any given moment. I’m still unsure exactly why they chose to do which test in each instance – in table 5 they appeared to switch between a Wilcox and a t-test at random. Although in table 3 they appear to have used a Uncorrected Chi squared based on the footnote, I’m not exactly sure, based on how one could be constructed with different expected values, if this was appropriate. No statistical expert am I, but again this smells a bit like statistical fishing to me. Even so, it doesn’t change the relevance of the results. Even if it does technically pass statistical muster, it’s still just the first step in a real scientific investigation. Another GMO expert suggests given the levels of mold they measured on their GM corn, it could have been a result of their source selling them moldy feed (at levels much higher than are usually found on GM crops).

So, to summarize, in this paper the authors performed a large non-specific screen for potential evidence of harm from GM crops. Of the many analyses performed, one showed statistical significance for severe stomach inflammation on a pathology scale in the GM group, but this effect rapidly-disappears if one groups inflammation based on broader categories. The clinical significance of this finding can only be determined by subsequent hypothesis driven research into this potential effect, but it is equally likely this is a result of random chance.

Or you can skip all the words above and read the XKCD one of Mark Lynas’ commenters suggests

XKCD knows stats

A final note, I’m not interested in comments saying I work for Monsanto, that I’m a corporate shill, blah blah blah. I haven’t worked for, or accepted money from, a corporation in my adult life (excluding Nat Geo sending me beer money for this blog, and working as a valet for Toyota dealership when I was 16). Address the data, the paper, relevant biological arguments etc, or get lost.

**Update**
In reading an additional response to the Carman et. al study, I now change my opinion on this paper from “competently performed but meaningless” to “totally meaningless”.

At issue is a criticism by Robert Friendship in the link above, that the author’s assay for inflammation is basically meaningless. In my initial read of the paper I didn’t notice this sentence “Typical examples of each of the four categories of inflammation are shown in Figure 1. For a severe level of inflammation, almost the whole fundus had to
be swollen and cherry-red in colour.”

I incorrectly assumed the authors had taken sections, performed histology, then assessed inflammation based on a legitimate pathological scale. This was apparently too generous. No, they just looked at the color of the stomach by gross pathology. As Dr. Friendship points out, this is meaningless.

No, It's Not the Sugar – Bittman and MotherJones have overinterpreted another study

Diet seems to be all over the New York Times this week, with an oversell of the benefits of the Mediterranean diet, and now Mark Bittman, everyone’s favorite food scold, declaring sugar is the culprit for rising diabetes. His article is based on this interesting new article in PLoS One and begins with this wildly-inaccurate summary:

Sugar is indeed toxic. It may not be the only problem with the Standard American Diet, but it’s fast becoming clear that it’s the major one.

A study published in the Feb. 27 issue of the journal PLoS One links increased consumption of sugar with increased rates of diabetes by examining the data on sugar availability and the rate of diabetes in 175 countries over the past decade. And after accounting for many other factors, the researchers found that increased sugar in a population’s food supply was linked to higher diabetes rates independent of rates of obesity.

In other words, according to this study, obesity doesn’t cause diabetes: sugar does.

No! Not even close. I hate to repeat his misstatement, because I’d hate to reinforce this as a new myth, but it’s critical to see his full mistake here. This is a wildly inaccurate summary of the authors’ findings, and one they don’t even endorse in their discussion. Bittman has actually just said “obesity doesn’t cause diabetes”, and now has proven himself a deluded fool.

Let’s talk about this paper. This is what is called an “ecological study”, which means it studies populations as a whole, rather than individual patients. Using data from the United Nations Food and Agricultural Organization, the International Diabetes Federation, and various economic indicators from the World Bank, the authors compared populations of whole countries, in particular the prevalence of diabetes correlated to other factors such as GDP, urbanization, age, obesity, and availability of certain varieties of food like sugar, meat, fibers, cereals and oil. Using the rise, or fall, of diabetes prevalence over the last decade in various countries, they correlated this increase with increasing availability of sugar, obesity, urbanization, aging populations etc., and found a few interesting things. For one, increases in GDP, overweight and obesity, tracked significantly with increasing diabetes prevalence. But interestingly, when those factors were controlled for, increasing availability of sugar also tracked linearly with increasing diabetes prevalence, and the longer the duration of the exposure, the worse it got.

However, this does not mean that “obesity doesn’t cause” diabetes, if anything, it’s further support for the exact opposite. While a correlative study can’t be a “smoking gun” for anything, the data in this paper supports increasing modernization/GDP, obesity, and sugar availability are all correlated with higher diabetes prevalence. Even if the sugar relationship is causal, which is no guarantee, the increase in sugar availability could only explain 1/4 of the increase in diabetes prevalence. Obesity is still the main cause of diabetes, which can be demonstrated on an individual level by increases in weight resulting in loss of glycemic control, and subsequent weight loss results in return of euglycemia. In particular, the results of studies of bariatric surgery, in both restrictive and bypass procedures, weight loss is accompanied by improvement in diabetes. The attempts of toxin paranoids like Bittman to reclassify sugar as a diabetes-causing agent, and to dismiss obesity as a cause, are highly premature.

Mother Jones, has a slightly more balanced read, but it still oversells the results.

This is a correlation, of course, and correlation does not always equal causation. On the other hand, it’s an exceptionally strong correlation.

Well, that’s another overstatement. Want to see a picture?

http://www.plosone.org/article/fetchObject.action?uri=info:doi/10.1371/journal.pone.0057873.g002&representation=PNG_M

Article Source: The Relationship of Sugar to Population-Level Diabetes Prevalence: An Econometric Analysis of Repeated Cross-Sectional Data

Basu S, Yoffe P, Hills N, Lustig RH (2013) The Relationship of Sugar to Population-Level Diabetes Prevalence: An Econometric Analysis of Repeated Cross-Sectional Data. PLoS ONE 8(2): e57873. doi:10.1371/journal.pone.0057873

Figure 2. Adjusted association of sugar availability (kcal/person/day) with diabetes prevalence (% adults 20–79 years old).

I wonder what the R-squared is on that line fit. Now, consider a comparison with obesity rates by diabetes prevalence:

Figure 1. Relationship between obesity and diabetes prevalence rates worldwide.

Obesity prevalence is defined as the percentage of the population aged 15 to 100 years old with body mass index greater than or equal to 30 kg/meters squared, from the World Health Organization Global Infobase 2012 edition. Diabetes prevalence is defined as the percentage of the population aged 20 to 79 years old with diabetes, from the International Diabetes Federation Diabetes Atlas 2011 edition. Three-letter codes are ISO standard codes for country names.
doi:10.1371/journal.pone.0057873.g001

Hmm, they didn’t fit a line here, but I can bet the fit would be better. Diabetes strongly correlates with BMI, this has been shown time and again using national survey data like NHANES or SHIELD. And before people start whining about BMI as an imperfect measure of obesity, it is perfectly appropriate for studies at a population level, and other metrics such as waist size, hip/waist ratios etc., all show the same thing. Diabetes risk increases linearly with BMI, with as many as 30% of people with BMI > 40 having diabetes, and further, we know from cohort and interventional studies that weight loss results in decreased diabetes. Much of this data is correlative as well (with the exception of the weight-loss studies), and the study that would prove this for certain – dividing people into diets providing excess fat, vs sugar, vs mixed calories, vs controls, with resultant measurement of diabetes rates, would be unethical. Either way, declaring sugar the enemy is both incomplete, and premature. While this paper provides interesting correlative evidence for increased sugar availability increasing diabetes prevalence, it is still subject to risk of confounding errors, it is correlative, and the link does not explain away other known causes of type II diabetes such as obesity. It is a warning however, and we should dedicate more study towards determining if sugar consumption (rather than mere availability) is an independent risk factor for type II diabetes.

Bittman has wildly overstated the case made by this article. He should retract his claims, and the title and false claims should be corrected by the editors. This is a terrible misrepresentation of what this study shows.