Peer Review – Where you thought it ended? That’s just the beginning!

plato

“This is a peer-reviewed study!”

In the increasingly heated battles waged lately by crusaders against innovation in agriculture, such assertions are increasingly thrown down like a gauntlet. The intent is to negate findings by regulators and scientists around the world that crops and foods improved through biotechnology are safe. These advocates argue passionately that “paper X”, published in a scientific journal after being reviewed by anonymous scientists and an editor, is sufficient to overturn the findings of hundreds of previously published reports (see http://www.biofortified.org/genera/guide/) to say nothing of the vast experience accumulated through the consumption of trillions of meals derived from biotech improved crops since they first entered the marketplace in the mid 1990s. When these papers are criticized by scientists post-publication, cries of censorship and persecution inevitably arise, and are routinely coupled with claims that the critics are bought and paid for by vested corporate interests (see  http://www.infiniteunknown.net/2012/12/17/smelling-a-corporate-rat-%E2%80%93-the-move-to-suppress-seralini-gmo-study/ and http://www.globalresearch.ca/gmo-researchers-attacked-evidence-denied-and-a-population-at-risk/5305324). But the noisemakers overlook something fundamental about the culture of science:  where they thought peer review ended -is really where it gets going.

Peer Review – What is it?

Like so much of Western thought, the idea of peer review has its roots in Plato’s dialogues. Following Socrates, it’s all about asking questions, challenging and re-examining presuppositions, and following the data. To that end, credible scientific journals follow a general pattern: when an author submits a manuscript for possible publication, editors solicit several of the author’s peers (the canonical number is three, but it can vary) to review the paper. These “peer reviewers” examine the methodology, data, and reasoning of the authors. Their charge is to make sure the experimental design is sound; the data produced actually address the hypothesis being tested; that previous related work is acknowledged, considered, and cited; and that the conclusions the author presents in the paper follow from the data.  Deficiencies against any of these criteria weigh against the proposal to publish, though if deficiencies identified are minor, or can be addressed fairly easily, authors are often asked to revise as a pre-condition for publication.

Papers that make it through this gauntlet and are then published are said to be “peer reviewed.”  But this does not mean they are now graven in stone, immune to challenge. What it means is that they have passed the first hurdle, and are now fit to be shared with the global scientific community, which gets to check the work of the “peer reviewers.” And sometimes they make mistakes. Let’s consider a recent example.

 

A paper by French scientist Gilles-Eric Séralini[i] that appeared in a journal published by Elsevier, a prominent scientific publisher, claimed that the long-term consumption by rats of biotech improved maize and (separately) glyphosate (the popular herbicide Roundup) produced an excess of tumors in rats. The paper is claimed by many biotech opponents to be the first long-term study of the safety of “gmos” (genetically modified organisms).  It was not. The authors claimed it showed long-term consumption by rats of biotech improved maize and (separately) glyphosate (the popular herbicide Roundup) produced an excess of tumors in rats.  They buttressed the claim with lurid photos of rats with gigantic tumors and their findings were quickly picked up by activists as a smoking gun against genetically modified organisms (GMOs). But there were more than a few problems with the paper.  I won’t re-hash them all here, but they’ve been pointed out in an unprecedented avalanche of letters to the editor of the journal,  critical reviews by official regulatory bodies, critical media coverage  and savaged in the blogosphere.  Of course, from the beginning, it didn’t take a sensitive nose to smell a rat, as the embargoed press release   announcing the paper only to journalists who had to promise to abide by unprecedented constraints designed to minimize the potential for negative media coverage.  The authors refused to release the full description of materials and methods supposedly used in the study and falsely declared they had no conflicts of interest (despite the fact that funding was obtained, at least in part, by commercial entities that benefit from anti-GMO fears). Second, the authors violated basic principles of animal care, which call for euthanizing animals to avoid unnecessary suffering once a clinical endpoint is reached, i.e., visible tumors as opposed to the gross deformities featured in the pictures accompanying the paper. Third, the strain of rats used in the study was one created by researchers to use in testing of chemicals for carcinogenicity, and about 80% of these rats would be expected to develop tumors  over their lifespan no matter what they are fed… which happens to be precisely what the authors report in this paper for both control and experimental animals. The authors further failed to apply any appropriate statistical analysis of their results, and somehow missed noticing that slightly more of their control animals developed tumors than did those fed glyphosate or biotech maize.  In other words, based on their own data, it would seem the biotech maize actually reduced the incidence of tumors compared to controls.[ii]

In summary, the global scientific community of peers reviewed the published paper and reached a clear consensus position, as captured in this statement from the European Food Safety Authority[iii]:

The… authors’ conclusions cannot be regarded as scientifically sound because of inadequacies in the design, reporting and analysis of the study as outlined in the paper. Consequently, it is not possible to draw valid conclusions about the occurrence of tumours in the rats tested. Based on the information published by Séralini et al., EFSA finds there is no need to re-examine its previous safety evaluations of NK603 or to consider these findings in the ongoing assessment of glyphosate.

The failings of the paper are, indeed, so numerous and flagrant as to lead some to charge the authors with “gross scientific misconduct and attempted fraud .”

That’s how peer review works.

There are many ways peer review can go awry, and let an unworthy paper slip into print. Peers can fail to understand or appreciate intricacies of the experimental design, or miss that the wrong statistical tools were used to analyze the results, or the right tools used in the wrong way. This is why three peers are most often consulted – it is hoped, and usually the case, that substantial mistakes will be caught by at least one of the three pairs of eyes. But if the peers are not well chosen, the potential for errors increases.

How are the peers selected?  This is perhaps the Achilles heel of peer review. Reviewers must be individuals versed in the subject matter that is the focus of the paper. If they are not familiar with the field, the value of their opinions and their ability to offer critical insights is reduced; the odds are increased that something will escape notice. And they must be willing to work for free – nobody gets paid to peer-review a paper; and they must be willing to toil in anonymity – they get no credit, except perhaps informal good will with the editor. It is the journal editor’s job to find the right peers to review a paper. So we can see there are a number of perverse incentives in play:  if a peer does a poor job, he/she is at little risk of exposure. To decline a request to review carries some stigma, but the anonymity of peer review limits this, and there is little clear penalty for peers who do not give the task their full attention. But this is why three peers are usually tasked with reviewing a paper. Anonymity also provides a measure of reassurance – it allows a scientist to offer pointed comments without fear of incurring the wrath of a powerful author, or embarrassing a colleague if pointing out a basic mistake – all to the good. But despite all these safeguards, sometimes peers just miss the boat; a paper with significant flaws in methodology or reasoning makes it into print even though it is so defective it should never have seen the light of day.

For this reason scientists (as opposed to propagandists) view publication as the beginning of peer review, rather than the end. The success and the value of the process depends on the participation of the global scientific community. Papers that survive this scrutiny help to map new approaches to problem solving in particular fields in the endlessly iterative efforts of scientists to more clearly understand the world. As Winston Churchill said of democracy, so it is with peer review:  “It’s absolutely the worst form of government [or, in this case, quality assurance], except for all the others.”



[i] Gilles-Eric Séralinia, Emilie Claira, Robin Mesnagea, Steeve Gressa, Nicolas Defargea, Manuela Malatestab, Didier Hennequinc, Joël Spiroux de Vendômoisa.  2012.  Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize.  Food and Chemical Toxicology 50 (11) November; 4221–4231.

[ii] Not to worry – the differences were not statistically significant.

Print Friendly

About the author

L. Val Giddings has nearly three decades of experience in science and regulatory policy relating to biotechnology innovations in agriculture and biomedicine. He works with ITIF to bring intellectual leadership to examination of the constraints inhibiting innovations in these areas, and devising remedies to those constraints.
  • Culture Shift

    Peer review is the only gold standard for scientific truth and each such publication helps everyone get closer to that truth. The only thing that can refute the conclusions of Seralini’s publications is another peer reviewed publication that contradicts those conclusions. Nothing you can otherwise say about it is really relevant except to also tell the truth about the industry ties of everyone like you with industry ties who, like you, try by every means to cast doubt on the conclusions of the papers evidencing health risks of GMOs.

  • Andy

    I most definitely agree. Lets have the bio-tech scientists with industry ties stop the onslaught of condemning anything negative said about GMO’s via replication. (Instead of obnoxious taunts and the systematic ganging up on their fellow scientists) Any design can be ripped to shreds and what they are doing is not science. It is pseudoscience. Replication is key.

  • Dave Wood

    I think you have quite missed the point of of the article. Peer review is the first stage in a sifting process. By chance it is possible to get two or three dud peer reviewers and a published paper in a low-quality journal. Then anyone can pile in as a reviewer/critic – as happened with Seralini’s paper. Scientists do not refute `conclusions’ – they refute hypotheses. Seralini tried to prove (not disprove) an unlikely combination: that both a herbicide and a plant not containing the herbicide were equally damaging to rats (that just happened to be very tumour-prone: that is what they were selected for). And what has industry to do with it? There is a vast organic farming industry that tries to junk anything to do with GM crops. Industry – aka as Monsanto – gets it in the neck for exporting North American GM farming technology to the competition in South America and India – to the great cost of North American crop exporters – a multi-billion dollar industry.

  • Valentine Dyall

    But, of course, all those many critical comments by scientists, and by many academies of science and regulatory bodies around the world, are in themselves further peer reviews. Culture Shift has responded exactly in line with expectation: using the industry tie smear instead of argument. He and his allies do not learn and will not change their verbiage because they do not see it as being in their interest to do so. As soon as you read the industry tie argument you know you have a propaganda statement not a reasoned approach.

  • http://www.perelman.net/ Lewis J. Perelman

    Contrary to the claim here by “Culture Shift,” Giddings if anything understates how flawed and unreliable the conventional peer review process is.

    An incisive critique of peer review is provided by Richard Smith, former editor of the British Medical Journal:

    “My point is that peer review is impossible to define in operational terms (an operational definition is one whereby if 50 of us looked at the same process we could all agree most of the time whether or not it was peer review). Peer review is thus like poetry, love, or justice. But it is something to do with a grant application or a paper being scrutinized by a third party—who is neither the author nor the person making a judgement on whether a grant should be given or a paper published. But who is a peer? Somebody doing exactly the same kind of research (in which case he or she is probably a direct competitor)? Somebody in the same discipline? Somebody who is an expert on methodology? And what is review? Somebody saying `The paper looks all right to me’, which is sadly what peer review sometimes seems to be. Or somebody pouring all over the paper, asking for raw data, repeating analyses, checking all the references, and making detailed suggestions for improvement? Such a review is vanishingly rare….

    “But does peer review `work’ at all? A systematic review of all the available evidence on peer review concluded that `the practice of peer review is based on faith in its effects, rather than on facts’.” (http://j.mp/15Gl8fu)

    Evidence of the unreliability of peer-reviewed publications has been provided by the work of Dr. John Ioannidis. His survey of articles published in peer-reviewed medical science journals found that 80-90% contained errors of data, analysis, or interpretation. (http://j.mp/10fZyvn)

    It is not unreasonable to wonder whether a researcher’s dependence on corporate sponsorship may bias his/her work. But it does not follow that the source of sponsorship necessarily invalidates or skews research findings. Moreover, as Giddings aptly points out, critics’ views are no less likely to be biased by their own sources of sponsorship, whether those are commercial competitors or groups/organizations with a particular political agenda.

    As Giddings suggests, the real ‘gold standard’ for validating scientific hypotheses and research findings is the substance of the research itself, its methodology and facts, as tested by replication and exposed via public criticism and debate.

  • No

    Yeah because every scientist that is pro-GMO has industry ties… I hope you spot the fallacy in there.

  • Paul Skvorc

    ONE – of several conceptual and logical flaws in the article – is: “It’s absolutely the worst form of government [or, in this case, quality assurance], except for all the others.”

    This comment treats a genuine problem as if there are only diametrically opposed options. In this case, either we have “peer review” as it currently exists, or “chaos”. A HUGE step in the right direction would be to eliminate that part of the current peer review process what FACILITATES deceit, the so-called “blind” review. We all know that is a complete farce. A second important step, would be to PUBLISH the review comments, with the reviewer’s contact information.

    Implementation of the above would eliminate both “you scratch my back and I’ll scratch yours”, and “you’ll never publish anything if I have anything to say about it”. Furthermore, it would reveal to the harsh ‘light of day’ the ‘unholy’ relationships that pervade the scientific community.