Advertisement

Ethics of reviewing scientific publications

Published:December 27, 2016DOI:https://doi.org/10.1016/j.ejim.2016.12.011

      Abstract

      Introduction

      The approval or rejection of scientific publications can have important consequences for scientific knowledge, so considerable responsibility lies on those who have to assess or review them. Today it seems that the peer review process, far from being considered an outdated system to be abandoned, is experiencing a new upturn.

      Aim and methods

      This article proposes criteria for the conduct of reviewers and of those who select them. While commenting on new emerging models, it provides practical recommendations for improving the peer-review system, like strengthening the role of guidelines and training and supporting reviewers.

      Conclusions

      The process of peer review is changing, it is getting more open and collaborative, but those same ethical principles which guided it from its very origin should remain untouched and be firmly consolidated. The paper highlights how the ethics of reviewing scientific publications is needed now more than ever, in particular with regard to competence, conflict of interest, willingness to discuss decisions, complete transparency and integrity.

      Keywords

      1. The debate on peer review: back to the future

      Giambattista Vico, the renowned Italian philosopher and historian of the age of enlightenment, expressed in his greatest work The New Science his belief in the recurrence of historical cycles, which could explain progress and civilization [
      • Vico G.
      New science. 1744. Cited in: Verene DP.
      ]. If we were to consider the evolution of the peer review system from its origins, we might dare to say that a cycle is ending and is being replaced by a new one based on the original ethical principles.
      The innovative method that guaranteed to the Royal Society of London, at the beginning of the nineteenth century, the scientific quality of the Philosophical Transactions (by which eminent scholars were asked to read and comment on the papers before they were published) [
      • Csiszar A.
      Peer review troubled from the start.
      ], spread throughout the scientific community as this took shape around the first scholarly journals. This system of evaluation by peers became a keystone of the whole publishing process, even if not as quickly as one might have expected [
      • Aileen F.
      Peer review: not as old as you might think.
      ]. In fact, scientific publishing has a long history, but the introduction of a formal peer review for submitted articles by external academics is relatively new. Science and JAMA, for example, introduced it in the 1940s, and Nature in 1967. However, soon after an initial appreciation, the system was put under severe scrutiny, and its many weaknesses and limits revealed. More than forty years ago the then-Editor of The New England Journal of Medicine, Franz Ingelfinger [
      • Ingelfinger F.J.
      Peer review in biomedical publication.
      ], posed two questions: Does peer review ensure that journals make good decisions about what to publish? Is it worth the price? The first International Congress on Peer Review and Biomedical Publication (PRC) was organized in 1989 to “subject the editorial review process to some of the rigorous scrutiny that editors and reviewers demand of the scientists whose work they are assessing” [
      • Rennie D.
      Preface.
      ]. Despite the fact that peer review was still considered the best possible system, it proved to be frequently unreliable and this was confirmed by the increase in publication frauds, followed by inevitable retractions. Winston Churchill's famous quote on democracy, considered as the worst form of government except for all the others, was often cited at the time. The system began to be seen as an inescapable process to undergo in order to obtain a grant or have a research published [
      • Wager E.
      • Godlee F.
      • Jefferson T.
      How to survive peer review.
      ] and was even mocked in a Christmas version of snakes and ladders [
      • Jefferson T.
      Snakes, ladders, and spin. Get peered.
      ]. The process was subjected to a long list of complaints to advocate its possible revolution [
      • Csiszar A.
      Peer review troubled from the start.
      ,
      ], in accordance with the many changes that were already disrupting the entire publication process [
      • Alleva E.
      • Napolitani F.
      Editorial a busy time for science editors.
      ].
      Today it seems that the peer review process, far from being considered an outdated system to be abandoned, is experiencing a new upturn. D. Rennie in a recent Comment on Nature calls for a more “scientific” peer review, adherent to the specific guidelines issued internationally for research reporting, “journals must accept and promote these guidelines and ensure that reviewers hold authors to them” [
      • Rennie D.
      Let's make peer review scientific.
      ]; a more rigorous application of the method by the academic community and a more scrupulous control of the process at editorial level has been seen as a possible way to avoid cases of “organised crime against academic peer review” [
      • Cohen A.
      • Pattanaik S.
      • Kumar P.
      • Bies R.R.
      • de Boer A.
      • Ferro A.
      • et al.
      Organised crime against the academic peer review system. Editorial.
      ].

      2. Preserving ethical principles while improving peer review procedures

      The process of peer review is changing, it is getting more open and collaborative, but those same ethical principles which guided it from its very origin should remain untouched and be firmly consolidated: the reviewers' moral integrity, the transparency, the responsibility and the profound accuracy (which requires time and effort) used in judging the research works and reporting of their peers. As COPE Ethical Guidelines for Peer Reviewers [
      • Committee On Publication Ethics (COPE)
      Ethical guidelines for peer reviewers.
      ] state, “the process depends to a large extent on trust, and requires that everyone involved behaves responsibly and ethically.”
      Several authoritative institutions, in the years, developed guidelines and codes. The Code of conduct and best practice guidelines for journal editors [
      • Committee on Publication Ethics (COPE)
      Code of conduct and best practice guidelines for journal editors.
      ] by the Committee on Publication Ethics (COPE) and the Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals [
      • International Committee of Medical Journal Editors (ICMJE)
      Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals.
      ] by the International Committee of Medical Journal Editors (ICMJE) are landmarks, but other relevant documents also exist. For example, in the “Recommendations on publication ethics policies for medical journals” [
      • World Association of Medical Editors (WAME)
      Recommendations on publication ethics policies for medical journals.
      ] the World Association of Medical Editors (WAME) suggests a list of major elements of a high-quality review. Several lists of requirements that reviewers should meet have also been proposed. In the Encyclopedia of Global Bioethics Ana Borovecki suggests a series of conditions [
      • Borovecki A.
      Publication ethics.
      ] that echoes the WAME Recommendations.
      However, an analysis of key documents and recommendations in the realm of publication ethics is not the aim of this note. Rather, it is the thought and consideration of a bioethicist, of an editor and of a pharmacologist, with many years of experience in their fields, about which possible topics could be further discussed, which practical opportunities could be seized to secure the peer-review ethical principles against possible threats which might come in science publishing when many other traditional boundaries are crossed. Reinforcing ethical implications is crucial not only for evaluating research, but for its conduction, productivity, dissemination and translation into practical benefits for the patients and for the community.

      2.1 Open peer review and open access

      Traditionally the peer review is a closed system. Most of the journals who adopted it in the first half of the 20th century used a single/semi-blind peer review by which reviewers knew the names of the authors of the submitted paper but not vice versa. The reviewers' identity was rendered anonymous. In addition, their opinions were only known by the editors and by the authors, but nobody else could access these comments. The benefits and drawbacks, pros and cons, biases and other inconveniences of this system have been extensively discussed and reported in the literature while a call for more openness started to bring its first results. The names of the reviewers started to be disclosed to authors, and the reviewers' comments to be made available and archived along with the manuscript. More recently, several journals operate a totally open peer review system, posting online the entire pre-publication history of a single paper, including decision letters, different versions of the manuscript and other reports. Is that so simple?
      At a first glance, this progressive opening of the peer review process seemed to be naturally occurring, concomitantly with the spreading and the development of the Open Access movement calling for an access free from all restrictions to each form of research output (February 2002, Budapest). But things did not really go as one might have expected. Shifting the cost of publications from the readers (libraries subscriptions) to the authors (publications fees or articles processing charges), has quickly generated a market of profit-maker publishers (so-called predatory publishers) and a proliferation of journals of scarce scientific quality which together caused an exponential increase in the submissions. While the Beall's list of potential, possible, or probable predatory scholarly open-access publishers is getting longer, ethical policies and code of practices should be strengthened and it is necessary to raise awareness of the danger of allowing the proliferation of such an unreliable literature in a delicate field like public health.

      2.2 Strengthening the role of guidelines

      COPE Guidelines for reviewers are very comprehensive, they “set out the basic principles and standards to which all peer reviewers should adhere during the peer-review process”, still, the cases being brought to COPE are getting more complex and require an increasing number of category classifications to cover all the issues [
      • Hames I.
      • Pierson C.A.
      • Ridgeway N.E.
      • Barbour V.
      Publication Ethics: 16 years of COPE.
      ].
      As things get more complex in science editing, more recommendations are needed, as confirmed by the long list of guidelines under development in the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network website, which lists essential resources for writing and publishing health research (http://www.equator-network.org/library/reporting-guidelines-under-development/). More than 300 reporting guidelines are included in the EQUATOR website, at present, in different clinical areas and for different study types. As new types of research articles stand out among traditional ones, further guidance is needed, not only for authors but also for reviewers. See for instance the recently published paper providing a number of recommendations for peer reviewers of narrative literature reviews [
      • Byrne J.A.
      Improving the peer review of narrative literature reviews.
      ].

      2.3 Post-publication and collaborative peer-review

      In addition to pre-publication peer-review, a new system of post-publication peer-review is becoming widespread. Electronic publishing has allowed “users” not only to download but to comment, review and even rate the published papers, which were accepted through a traditional peer review process and are now subject to further comments. This system is offered by many electronic journals but also by databases. In 2013, the NIH announced the launch of the pilot version of “PubMed Commons” which offers the possibility to comment on all papers cited in PubMed database (https://www.ncbi.nlm.nih.gov/pubmedcommons/). Presented as “A forum for scientific discourse” it is open to all authors of publications indexed in PubMed who are eligible to join and post comments on any citation.
      All the stakeholders involved in the process (researchers, editors, reviewers, but also the general public, the sponsors, the patients, and the policymakers) should take full advantage of the extraordinary opportunities offered by technologies not only to improve the quality and to accelerate the dissemination of scientific knowledge, but to assess it in an open and collaborative way. Open Access journals are offering new assessment methods in the form of post publication comments, social media or collaborative evaluations. Some open access publishers, or groups of publishers, are providing a “portable peer-review”: they offer authors to rapidly transfer a rejected submission to another of their lower impact journals [
      • Björk B.-C.
      • Hedlund T.
      Emerging new methods of peer review in scholarly journals.
      ].
      However diversified this scenario can be, the emphasis, above all, is again on ethical issues. The need to discuss these aspects is confirmed by the founding in 2015 of in a new open access journal Research Integrity and Peer Review (https://researchintegrityjournal.biomedcentral.com) which is entirely devoted to all aspects of integrity in research publication including peer review; and also by the great interest and success of the International Congress on Peer Review and Scientific Publication. In September 2017 in Chicago a great variety of topics related to publication ethics, integrity and credibility of peer review will be discussed (http://www.peerreviewcongress.org/index.html).

      2.4 Patients: welcome in the peer review process

      In the last decades, patients gradually entered a world traditionally confined to doctors, medical professionals, scientists and other specialists and experts. They first approached it as simple users seeking health information through the new systems available online. The US National Library of Medicine had perceived this new interest by the patients and, at the beginning of the 21st century, formally declared that it would “seek to serve the general public after over a century of serving the library and medical communities exclusively” [
      • Lacroix E.M.
      • Mehnert R.
      The US National Library of Medicine in the 21st century: expanding collections, nontraditional formats, new audiences.
      ]. In the online age which was about to explode, at the time, “searching” became part of the patients' consciousness and the US NLM, in response to this need, offered to patients, families and the general public a powerful tool like MedlinePlus, introduced in 1998 free to all. The patient became an “informed patient” who, as expected, asked to get more involved in decisions concerning his/her own health, thus beginning a partnership with medical doctors which, after about ten years, will be recognised as capable of effectively improving healthcare “Patients can improve healthcare: it's time to take partnership seriously” [
      • Richards T.R.
      • Montori V.M.
      • Godlee F.
      • Lapsley P.
      • Paul D.
      Let the patient revolution begin. Editorial.
      ].
      This is part of the so called “patient revolution”. Though patients will never have the required scientific competencies to review complex research papers, “We need to accept that expertise in health and illness lies outside as much as inside medical circles” [
      • Richards T.R.
      • Montori V.M.
      • Godlee F.
      • Lapsley P.
      • Paul D.
      Let the patient revolution begin. Editorial.
      ]. Another step forward, the patients are getting involved in roles once exercised by an elitist community and finally they entered the sacred world of science journals. “Now we want to develop a strategy for patient partnership that will be reflected across the entire journal. We plan to establish a panel of patients and clinicians to help us with this work and will report back on our progress”. Patients are invited to be part of the peer review process and review articles alongside other peer reviewers [
      • Richards T.R.
      • Montori V.M.
      • Godlee F.
      • Lapsley P.
      • Paul D.
      Let the patient revolution begin. Editorial.
      ,
      Guidance for BMJ patient reviewers. Patient peer review at the BMJ.
      ].
      The impact of this additional peer (or “critical”) review strategy will be evaluated, but in the meantime this practice will be extended to journals accepting papers on medical conditions, rare diseases etc. Editors should consider the opportunity to implement such a system in their editorial routine and new standards and guidelines for patients peer review should be further discussed among the scientific community [
      Guidelines for reporting the impact of patient and public involvement in research: strengthening the evidence base of patient and public involvement summary.
      ].

      2.5 Training and supporting reviewers

      Journals can improve review quality by training, rewarding and supporting reviewers (who are unpaid and overworked); by developing criteria for the selection of reviewers and by adopting changes that might improve ethical standards in peer review. About 77% of reviewers interviewed by Wiley in a 2015 survey expressed an interest in receiving further training [
      • Warne V.
      Rewarding reviewers – sense or sensibility? A Wiley study explained.
      ]. Younger peer-reviewers should be trained and enrolled in the process and be kept up-to-date in new approaches and methods of editing [
      • Byrne J.A.
      Improving the peer review of narrative literature reviews.
      ]. In particular,
      • 1)
        Reviewers should:
        • ­
          ask themselves if they really are competent to undertake the task assigned to them;
        • ­
          decide whether or not to accept an assignment only after carefully evaluating the adequacy of their own knowledge, supplemented where necessary with additional information that is readily available from reliable sources;
        • ­
          carefully consider the possibility of conflicts of interest, which may take various forms: friendship with the author(s) of the proposal/article, membership of the same academic or industrial institution, financial interests, competing in the same research areas -to mention only the most obvious factors that could jeopardize an objective assessment;
        • ­
          be prepared to consider original proposals or articles even where they concern theories or knowledge that might upset existing equilibria or challenge convictions that have matured over many years;
        • ­
          not allow themselves to give in to pressure or – even more important – to be accessories to deception. “Publish or perish” has become a form of extreme pressure and competition that can interfere with decision-making;
        • ­
          not take advantage of their role to delay research by competitors, for example by requesting the repetition of experiments or trials already concluded. This can simply lead to waste of resources;
        • ­
          be prepared to allow their names and review comments to be posted in a footnote to the approval of proposals or acceptance of articles for publication, in the interest of transparency, and to specify the reasons for their decisions;
        • ­
          be fully aware of the limits of their decisions (which may differ from, or even contrast with those of other reviewers) and of the value of their recommendations;
        • ­
          be aware that their job calls for a tightly balanced approach to decision-making and that it should be directed towards improving knowledge.
      • 2)
        Those who select them should be fully aware that their role carries considerable responsibility. For this reason:
        • ­
          criteria should be established to guide the selection process, bearing in mind, above all, the possibility of conflicts of interest;
        • ­
          where the selection process is entrusted to a committee, a balance of competences should be sought so that a sufficient variety of specialized skills is represented and “fashionable” opinions are not allowed to prevail;
        • ­
          in the case of articles for publication, the initial reviewers should complement one another: if an article concerns a controlled trial with diabetic patients, for instance, it should be reviewed not only by a clinician with expertise in diabetes but also by an expert in the conduction of controlled clinical trials;
        • ­
          the correct conduct of reviewers should be constantly monitored both on the basis of their work and by asking authors for their views; authors should not passively accept ill-considered reviews but cooperate with editors and others who select reviewers to improve the quality of the review process;
        • ­
          identities of whoever selects the reviewers should be indicated in a footnote to articles;
        • ­
          the reasons for refusing an article should be made known.

      3. Conclusions

      Many of the problems of the peer-review process come down to the poor quality of the peer-reviewers. Several authoritative documents list duties for editors and reviewers and a number of authors and institutions have suggested possible strategies to improve the methods of evaluations of journal articles. Some proposals are directed at improving the current procedures of peer review, while others suggest different methods. The arrival of internet calls for the replacement of traditional review procedures with an open evaluation [
      • Kriegeskorte N.
      Open evaluation: a vision for entirely transparent post-publication peer review and rating for science.
      ]. Many journals have already adopted fully open peer review procedures, post-publication evaluation and put online the reviewers' comments and their signatures, adopting precise rules for reviewers [].
      Its advantages include greater transparency, but the procedure also poses considerable risks for the quality of publications. Reviewers and those who select them, together with researchers, the public and sponsors, at international level, must all ask themselves how to avoid these risks and make a better use of new communications technologies to accelerate and improve the quality and dissemination of scientific knowledge. The scientific community must carry out its research on the basis of reliable and sound information – always bearing in mind the need to improve the clarity and intelligibility of knowledge as it trickles down to patients.
      In conclusion, the authors of this paper have attempted to provide some recommendations for improving the peer-review system, commenting on new emerging models which might become common practice in the future and that need to be further studied and assessed. They tried to highlight how the ethics of reviewing scientific publications is needed now more than ever.
      In fact, regardless of the future evolution of the reviewing system for research reporting, regardless of the innovations in the publishing and dissemination of research data, which are increasingly driven by open and collaborative experiences, a valid principle remains. The complex system of biomedical research rotates around a special hinge that holds the whole mechanism in place: the ethical and moral values of the people, whether they are researchers, reviewers, editors or even academics involved in the evaluation of scientists in their future careers.

      Conflict of interest

      The authors wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

      References

        • Vico G.
        New science. 1744. Cited in: Verene DP.
        Vico's New Science. A Philosophical Commentary. Cornwell University Press, Ithaca, NY2015
        • Csiszar A.
        Peer review troubled from the start.
        Nature. 2016; 532: 306-308
        • Aileen F.
        Peer review: not as old as you might think.
        Times higher education. June 25, 2015 ([Available online at: https://www.timeshighereducation.com/features/peer-review-not-old-you-might-think])
        • Ingelfinger F.J.
        Peer review in biomedical publication.
        Am J Med. 1974; 56: 686-692
        • Rennie D.
        Preface.
        in: Peer review in scientific publishing. Papers from the First International Congress on Peer Review and Biomedical Publication. Council of Biology Editors, Chicago, IL1991
        • Wager E.
        • Godlee F.
        • Jefferson T.
        How to survive peer review.
        BMJ Books, London2002
        • Jefferson T.
        Snakes, ladders, and spin. Get peered.
        BMJ. 2003; 327: 1439
      1. Nat Neurosci. 2005; 8: 397
        • Alleva E.
        • Napolitani F.
        Editorial a busy time for science editors.
        Ann Ist Super Sanita. 2013; 49: 1
        • Rennie D.
        Let's make peer review scientific.
        Nature. 2016; 535: 31-33
        • Cohen A.
        • Pattanaik S.
        • Kumar P.
        • Bies R.R.
        • de Boer A.
        • Ferro A.
        • et al.
        Organised crime against the academic peer review system. Editorial.
        Br J Clin Pharmacol. 2016; 81: 1012-1017
        • Committee On Publication Ethics (COPE)
        Ethical guidelines for peer reviewers.
        COPE, 2013 ([Available online at: http://publicationethics.org/files/Peer%20review%20guidelines_0.pdf])
        • Committee on Publication Ethics (COPE)
        Code of conduct and best practice guidelines for journal editors.
        March 7 2011 ([Available online at: http://publicationethics.org/files/Code%20of%20Conduct_2.pdf])
        • International Committee of Medical Journal Editors (ICMJE)
        Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals.
        2013 ([Available online at: www.icmje.org/icmje-recommendations.pdf])
        • World Association of Medical Editors (WAME)
        Recommendations on publication ethics policies for medical journals.
        (Available online at)
        • Borovecki A.
        Publication ethics.
        in: Ten Have H. Encyclopedia of global bioethics. vol. 3. Springer Reference – Business Media, Dordrecht2016: 2362-2368
        • Hames I.
        • Pierson C.A.
        • Ridgeway N.E.
        • Barbour V.
        Publication Ethics: 16 years of COPE.
        7th International Congress on Peer Review and Biomedical Publication. September 2013; : 8-10
        • Byrne J.A.
        Improving the peer review of narrative literature reviews.
        Res Integr Peer Rev. 2016; 1 ([https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-016-0019-2])https://doi.org/10.1186/s41073-016-0019-2
        • Björk B.-C.
        • Hedlund T.
        Emerging new methods of peer review in scholarly journals.
        Learned Publ. 2015; 28: 85-91
        • Lacroix E.M.
        • Mehnert R.
        The US National Library of Medicine in the 21st century: expanding collections, nontraditional formats, new audiences.
        Health Inf Libr J. 2002; 19: 126-132https://doi.org/10.1046/j.1471-1842.2002.00382.x
        • Richards T.R.
        • Montori V.M.
        • Godlee F.
        • Lapsley P.
        • Paul D.
        Let the patient revolution begin. Editorial.
        BMJ. 2013; 346: f2614
      2. Guidance for BMJ patient reviewers. Patient peer review at the BMJ.
        (Available from)
      3. Guidelines for reporting the impact of patient and public involvement in research: strengthening the evidence base of patient and public involvement summary.
        (Available from)
        • Warne V.
        Rewarding reviewers – sense or sensibility? A Wiley study explained.
        Learned Publ. 2016; 29: 41-50
        • Kriegeskorte N.
        Open evaluation: a vision for entirely transparent post-publication peer review and rating for science.
        Front Comput Neurosci. 2012; 6: 176-193
      4. British Medical Journal. Guidance for peer reviewers.
        (Available online at)

      Linked Article

      • Peer reviewing and Editor's decision should never be a personal conflicting matter
        European Journal of Internal MedicineVol. 40
        • Preview
          I came through the latest paper on the journal by Napolitani et al., about peer reviewing of scientific reports in the academic community of experts [1]. I joined the Editorial Board as a Member of some scientific journals, e.g. Nutrition (Elsevier), and the paper by Napolitani strongly attracted my interest. In particular, my attention was caught by a citation in the paper reference list, where the Editor in Chief of British Journal of Clinical Pharmacology, Prof. Adam Cohen, took a press charge against an editorial case where the process of peer reviewing was defiled by organized crime [2].
        • Full-Text
        • PDF