by Wendy Lipworth and Ian Kerridge*

Peer review of health-related manuscripts has enormous power in determining what is published in health-related journals, and what makes its way into health policy and clinical practice. However, peer review is at times ethically problematic and not always effective in achieving its goals. Over the past 25 years, a large number of debates about, and studies of, the peer review process has been published. Despite this, there is limited agreement about the strengths and weaknesses of peer review, and limited evidence about whether peer review achieves its goals and whether interventions to improve it have been successful. The authors argue that this state of affairs is not acceptable and that there is a need to systematise efforts to understand and improve the review process.

THE TROUBLE WITH HEALTH–RELATED “EVIDENCE”

Clinicians, public health practitioners and health policy-makers are constantly reminded that their practices should be “evidence-based” and that much of their current practice lacks sufficient scientific rigour. But while critics are quick to point out the failure of individual practitioners and policy-makers to incorporate evidence into their own practice, it is also recognised that the evidence presented to decision-makers – to a large extent in the form of “peer-reviewed” journal articles and practice guidelines based on published research – might itself be flawed. This might, in turn, be due to problems with the research carried out to generate evidence; problems with the production of practice guidelines; or problems with the process of publication in health-related journals, in particular the process of “peer review” through which decisions about publication are made.

CRITIQUES OF PEER REVIEW AND THE RISE OF “PUBLICATION ETHICS”

While many people believe that some form of pre-publication peer review is an important means by which the quality and integrity of published research can be promoted (a position the current authors support), it is also widely recognised that peer review is not perfect. In particular, reviewers are criticised for their failure to consistently ensure that only high-quality material is published, or to facilitate dissemination of important research. These problems were eloquently described by Stephen Lock, a past editor of the British Medical Journal, in his 1985 book entitled A Diffıcult Balance: Editorial Peer Review in Medicine.1 Richard Smith, also a past editor of the British Medical Journal, recently summed up his view of journal peer review as follows:2

“We have little evidence on the effectiveness of peer review, but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.”

Those who are critical of the peer review process frequently attribute its problems to unethical behaviour on the part of reviewers, who are seen to be concerned primarily with maintaining their own power and achieving their personal and communal aspirations, often at others’ (ie authors’) expense.3

Peer reviewers have been accused of many different forms of unethical practice. The most serious of these involve outright plagiarism or extreme bias due to conflicts of interest, but instances of perceived reviewer misconduct also include failure to disclose conflicts of interest, as well as more subtle misdeeds such as rudeness, deliberately delaying the publishing process, and failing to take the review process seriously. Editorial misconduct is also seen as a problem, with editors sometimes bypassing peer review, belittling competing authors, publishing their own work, or publishing work that supports sponsoring companies.4

The field of “publication ethics” (sometimes viewed as a branch of research ethics) continues to develop and there are now a number of organisations charged with overseeing publication in general, and peer review more specifically.5 The International Committee of Medical Journal Editors (ICMJE) is a group of medical editors who meet annually and have published a set of highly cited guidelines entitled “Uniform Requirements for Manuscripts Submitted to Biomedical Journals”. Of particular relevance to the integrity of the review process are statements relating to editorial freedom, editors’ and reviewers’ conflicts of interest, maintenance of both authors’ and reviewers’ confidentiality and prevention of publication biases such as positive result bias.6 The Committee on Publication Ethics (COPE), which was established in 1997, has examined over 250 cases in which there have appeared to be ethical problems with the publication process. These have included cases of reviewer and editorial misconduct such as editors-in-chief overriding external review in a manner suggestive of conflict of interest, and reviewers breaching authors’ confidentiality. COPE has also produced guidelines relating to peer and editorial review, including maintaining authors’ confidentiality; not plagiarising authors’ work; avoiding bias; and auditing editorial and review practices.7 Other bodies with an interest in publication ethics include the Council of Science Editors (CSE)8 and the World Association of Medical Editors (WAME).9 There is also an increasing number of academic and industry regulatory bodies aimed at maintaining scientific integrity and preventing scientific misconduct more generally, which include among their concerns the integrity of publication processes.10

THE CONTINUING NEED TO BETTER UNDERSTAND MANUSCRIPT REVIEW

In response to concerns about peer review, a number of practical strategies have been suggested and implemented including:

  • insisting that reviewers and editors declare their conflicts of interest, both financial and non-financial (epistemological, methodological, ideological, pedagogical);
  • allowing authors to exclude potentially biased and hostile reviewers;
  • having reviewers sign their reviews; and
  • publishing reviews for broader scrutiny.11

While these interventions have been the focus of empirical research, considerable uncertainty remains as to what journal peer review achieves and what, if any, interventions might improve the process.12 This lack of evidence has led several critics of peer review to claim that the peer review process is not sufficiently “scientifically” grounded, and that more research is needed. The authors of a recent Cochrane Review of peer review studies made particular note of the inadequacy of published research on peer review, and called for an extensive, multi-centre program of empirical research, cautioning:13

“Until such research is undertaken, peer review should be regarded as a long-standing, potentially expensive, untested process with uncertain outcomes.”

Despite such calls for editors to make evidence-based changes to their processes, comprehensive and systematic research into peer review has generally not eventuated. There are a number of possible explanations for this. First, most of the research into journal peer review has been carried out by working journal editors with limited time and resources and in an environment where collaboration between journals and the construction of large-scale research studies of peer review are extremely difficult. Secondly, insofar as there has been academic interest in journal peer review, there is no established academic discipline devoted to this topic. Instead of being concentrated in one domain, studies of peer review are dispersed through many disciplines, including clinical medicine, sociology of science and medicine, linguistics, history, bioethics, law and philosophy. Thirdly, it may be that the methods required to study peer review comprehensively, including in-depth social, scientific and qualitative methods, require skills that are relatively lacking in the health research community. And finally, it is possible that journal editors’ attention has shifted away from deep “insoluble” problems like peer review and towards more “pressing” issues such as researcher and author misconduct, including fraud.

This is not to say that editorial practices have been static. Some health-related journals, eg, have switched from anonymous peer review processes to a process of signed reviews. In the absence of good empirical data, there are many other reasons why journals might make such changes. Decisions could simply be pragmatic, based upon journals’ resources or upon external pressures on editors. Alternatively, editorial practices might reflect ethical, rather than empirical, considerations. For example, with respect to reviewer anonymity, while there is little positive or negative empirical evidence of any effect of blinding/unblinding on the quality, consistency, timeliness or outcomes of peer review,14 other arguments have been used to justify both anonymous and signed reviews. Those in favour of anonymous review argue that critical reviewers might fear, or actually experience, reprisal from disgruntled authors, while those in favour of signed reviews argue that allowing reviewers to remain anonymous insulates reviewers from accountability; that justice needs to be seen to be done; and that power without responsibility should not be tolerated.15 In this regard, it is noteworthy that, in relation to its own guidelines, the ICMJE makes it clear that:

“[t]hese recommendations are based largely on the shared experience of a moderate number of editors and authors, collected over many years, rather than on the results of methodical, planned investigation that aspires to be ‘evidence-based’”.16

WHERE TO FROM HERE?

It is clear, therefore, that despite several decades of debate and empirical research into journal peer review, efforts to improve the process still generally lack supporting evidence, and the justifications for particular editorial practices appear to be based primarily on pragmatic or ethical, rather than empirical, grounds. While understandable and well-intended, the current authors suggest that this state of affairs is problematic because we should know whether existing editorial practices (and changes therein) are effective and what, if any, practices might improve journal peer review.

This leaves us with a number of options. First, we could work to put into effect the multi-centre program of research suggested by the Cochrane reviewers17 and others18 so as to generate the systematic empirical evidence needed to improve journal peer review. Alternatively, we could leave journal peer review as it is, but acknowledge both the epistemological limitations of the process and its many social functions within research and professional communities (providing a means by which researchers can distinguish themselves professionally; strengthening academic communities; or building social movements around particular issues). Or we could work to supplement pre-publication peer review with other kinds of review, such as formalised post-publication criticism. These options are not mutually exclusive, and it seems likely that the best results may be achieved by some combination of revived research, modified expectations and supplementation of pre-publication review with other review mechanisms.

Whatever approach(es) are chosen, it is crucial that researchers, practitioners and policy-makers are cognisant of the ways in which their “evidence” comes into being and of the fact that, despite 25 years of effort, journal peer review processes are themselves not “evidence-based”. This is not to suggest that we should become cynical about peer review, but rather that systems of peer review be recognised as generally not being based upon firm scientific data and that they are as much about politics and ethics as about “evidence”.

*Wendy Lipworth, BSc (Med) Hons, MBBS, MSc, PhD, Postdoctoral Researcher, Centre for Values, Ethics and the Law in Medicine, University of Sydney, NSW, Australia, and Australian Institute of Health Innovation, University of New South Wales, NSW, Australia; Ian Kerridge, BMed (Hons), BA, MPhil (Cantab), FRCPA, FRACP, Director, Centre for Values, Ethics and the Law in Medicine, University of Sydney, NSW, Australia. Correspondence to: [email protected]; [email protected].
1 Lock S, A Diffıcult Balance: Editorial Peer Review in Medicine, Rock Carling Fellowship 1985 (Nuffield Provincial Hospitals Trust, London, 1985).
2 Smith R, The Trouble with Medical Journals (Royal Society of Medicine Press, London, 2006) p 89.
3 Atkinson M, “Regulation of Science by Peer-Review” (1994) 25 Stud Hist Philos Sci 147; Judson H, “Structural Transformations of the Sciences and the End of Peer Review” (1994) 272 JAMA 92; Godlee F, “The Ethics of Peer Review” in Hudson Jones A and McLellan F (eds), Ethical Issues in Biomedical Publication (Johns Hopkins University Press, Baltimore, 2000); Atkinson M, “‘Peer Review’ Culture” (2001) 7 Sci Eng Ethics 193.
4 Rennie D, “Misconduct and Journal Peer Review” in Godlee F and Jefferson T (eds), Peer Review in Health Sciences (BMJ Books, London, 2003); Wager E and Herxheimer A, “Peer Review and the Pharmaceutical Industry” in Godlee F and Jefferson T (eds), Peer Review in Health Sciences (BMJ Books, London, 2003).
5 McLellan F and Riis P, “Ethical Conduct for Reviewers of Grant Applications and Manuscripts” in Godlee and Jefferson, n 4; Horton R, “The Journal Ombudsperson: A Step Toward Scientific Press Oversight” (1998) 280 JAMA 298; Godlee, n 3.
6 International Committee of Medical Journal Editors, Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication (2007), http://www.icmje.org viewed 21 February 2011.
7 Committee on Publication Ethics, http://www.publicationethics.org.uk viewed 21 February 2011.
8 Council of Science Editors, http://www.councilscienceeditors.org viewed 21 February 2011.
9 World Association of Medical Editors, http://www.wame.org viewed 21 February 2011.
10 Office of Research Integrity, http://www.ori.dhhs.gov viewed 27 June 2010; Wager and Herxheimer, n 4.
11 Hojat M, Gonnella J and Caelleigh A, “Impartial Judgment by the ‘Gatekeepers’ of Science: Fallibility and Accountability in the Peer Review Process” (2003) 8 Adv Health Sci Educ Theory Prac 75; Kumashiro K, “Thinking Collaboratively About the Peer-review Process for Journal-article Publication” (2005) 75 Harvard Educ Rev 257; Bingham C, “Peer Review on the Internet: Are There Faster, Fairer, More Effective Methods of Peer Review?” in Godlee and Jefferson, n 4.
12 Jefferson T et al, “Editorial Peer Review for Improving the Quality of Reports of Biomedical Studies” (2007) Cochrane Database Syst Rev MR000016.
13 Jefferson, n 12, p 13.
14 Jefferson, n 12.
15 Godlee, n 3.
16 International Committee of Medical Journal Editors, n 6.
17 Jefferson, n 12.
18 Atkinson (2001), n 3.

The citation for this article is (2011) 18 JLM 724.