|Year : 2020 | Volume
| Issue : 1 | Page : 110-114
How to referee a paper - and save the world
Sanjay A Pai
Executive Editor, Indian J Cancer, Consultant Pathologist and Head of Pathology, Columbia Asia Referral Hospital Bangalore, Karnataka, India
|Date of Submission||20-Nov-2019|
|Date of Decision||23-Nov-2019|
|Date of Acceptance||04-Jan-2020|
|Date of Web Publication||26-Feb-2020|
Sanjay A Pai
Executive Editor, Indian J CancerConsultant Pathologist and Head of Pathology, Columbia Asia Referral Hospital Bangalore, Karnataka,
Source of Support: None, Conflict of Interest: None
|How to cite this article:|
Pai SA. How to referee a paper - and save the world. Indian J Cancer 2020;57:110-4
In the months that I have been editor of Indian J Cancer, I have met (virtually, pun absolutely unintended) several unenthusiastic referees. This is, of course, not an issue unique to this journal, but is a worldwide phenomenon. I recall reading a message in a group of the World Association of Medical Editors that 50% of those invited to review a paper, do not even accept the invitation. (The proportion at Indian Journal of Cancer is even higher). Even after accepting the invitation to review, the comments sometimes fall far short of expectation and are simply unhelpful. It often takes multiple reminders or even upto eternity (well, almost) for the comments to materialize.
Being asked to referee a paper is a mixed delight. It is surely an honour because it means that the editor thinks that you are capable of analyzing someone else's science. It implies you are an expert on the subject matter of the paper. You may add to your CV, the fact that you are a peer reviewer and by implication, an expert on the subject. The flip side, of course, is that despite being hard-pressed for time, you have to find time to review someone else's paper!
Refereeing a paper is an essential part of the academic cycle. It is an unpaid job; and you cannot even tell people that you helped select the paper that was awarded the Nobel prize, because you have to stay anonymous. Remember – the entire basis of peer review is this: you referee, gratis, for others and someone else referees, also gratis, for the papers that you write. Peer review has been around for over 350 years and often acts as a check on poor science. Don't break that cycle – else we would lose this important check. It is your duty to do this conscientiously and to contribute your bit to saving the world (of science). Here then, are some guidelines and suggestions for potential referees, based on others' as well as my experience.,, The neophyte would most definitely benefit from these tips, but even seasoned referees may learn a trick or two. These ideas are, of course, not cast in stone and you may quite rightly have your own style or variation. While much of this advice is generic and is applicable for most journals, a few suggestions are specific for Indian J Cancer.
Your duty is to advise the editor whether the paper should be accepted or rejected or needs to be sent back for revision. Of course, the editor may not follow your advice entirely and may take a final decision quite different from your suggestion.
Let me begin with what not to do and then get on to what to do.
How not to write the report
The most unhelpful referee reports are those along the lines of “Great paper, publish” or “Interesting” (a comment which is far more common than one would imagine) and its opposite, “…adds nothing new to the literature”, or a variant “Needs better discussion”. Please be specific. Why is the paper fantastic? What do you mean by “interesting?” State why you think that the paper does not add anything new to the existing literature. Refer to some of the papers which have addressed the subject previously. These papers may, often, be among the references that the authors have used in their paper. Failure to quote previous work may suggest either poor referencing on the part of the authors, or a deliberate attempt to not quote previous papers – both of which are inexcusable.
Basics – Before you start
If you have any conflict of interest in refereeing the paper, decline the invitation and do so immediately. Also, let the editor know the reason for this. The editor may still value your opinion, even after knowing about this conflict. Conflicts of interest may be financial or non-financial. The author (who you may recognize even in a blinded review) may be a collaborator or a foe, may be from your own department or from an institution where you have worked in the past, or – as I have seen in a case recently – a (current!) spouse.
Make sure you have the time to referee the paper in the time offered. If you simply cannot do it because you are, say, about to go on a vacation, reject the invitation. However, do not make a habit of this. You could ask the editor for some extra time, but if you do so, make sure that you stick to the promise and do not delay your report any further.
A late referee report may result in increased time to publication, resulting in delayed – or even lost – credit to the author, particularly if they are describing something novel. Further, the “hot” news has the potential to turn “cold” if you take an inordinate amount of time to review it.
Don't be what Paulus refers to as NERO – NEver REsponding, referring to referees who accept the invitation to referee but never respond thereafter. Similarly, do not be what I term (appropriately enough, in the context of NERO), FIDDLE - Frequent Inability To Deliver Decisions to Lamenting Editor. i.e. referees who raise many relevant queries on the initial submission – and then refuse to review the revised manuscript – thus leaving the editor in a no-man's land, if the editor is unable to evaluate the author's response herself/himself.
If you have recommended rejection of a particular paper previously for another journal, a few days ago, it may be advisable to refrain from reviewing the paper again because you are unlikely to be absolutely unbiased – and because the authors are unlikely to have made any changes in the short intervening period. If, however, considerable time has elapsed since you suggested rejection of the paper, such that it is possible that your comments would have been addressed, you may choose to review it. My preference, however, is to recuse because it may be unfair to the authors if the same person reviews every version of the paper.
Are you qualified to comment on the paper? Do you have the necessary understanding and expertise? Referees are sometimes selected from a PubMed search to identify similar papers. It is well possible that your contribution to that previous paper had nothing to do with the paper that you are being asked to referee.
Be familiar with the instructions to authors so that you can detect if the paper veers vastly off the journal style or ambit.
In case the journal does not follow a blinded review process and you recognize the author's identity and affiliation, you may realize that there are unstated conflicts of interest or potential biases on the part of the author that have not been stated. You must inform the editor about these issues.
Some general advice. Do not be sarcastic or rude. Usually, such comments would not reach the author, because the editor edits these words. However, you run the risk of getting into the editor's bad books for writing a potentially unusable review. This may not bode well when, subsequently, you submit a paper to the same journal. Also, if such comments escape the editor's eye and do reach the author, the review will show the journal, the editor and the unnamed referee (you!) in bad light. Do remember that the authors have in all probability given the paper their best effort. It may still not meet your exacting standards – but that is no reason to be uncivil. Your duty is to help weed out the bad material from the paper and seek out the message that the authors are trying to convey. Be objective in every way possible. Science is, after all, the search for the truth - and the truth is objective.
While typing in your comments on the online manuscript handling system, do note that many journals' software (including that of Indian J Cancer) automatically log out after varying periods of time and your comments may be lost in the ether – so it is best to work offline and then submit the comments. Alternatively, you could save your input frequently.
Evaluating the paper
Some of us are comfortable reading PDFs on screen. However, it is usually a better idea to print out the manuscript and annotate it while reading. At any rate, make it a point to jot down every relevant point as it crosses your mind while reading the paper – because it may not strike you the next time you look at the paper. Yes, you may need to look at the paper more than once, to offer useful comments.
“So what?” - This is the most important question to answer. If this is your reaction to a paper, it probably means that the paper does not merit publication. Many papers are merely a medical laundry list or may offer no new information.
Is there a major flaw in any part of the paper? The aim of the paper may be unimportant or the underlying hypothesis may be flawed. An ethics committee permission may not have been taken. The results may be way off the mark. The most common reason for rejection of a paper, however, is the use of inappropriate methods.
Almost every paper – even a very good one- has some flaws, albeit minor. The authors may have missed out on a recent or a classic publication on the subject and you may wish to point this out. Resist the temptation to suggest your own papers as references, unless you really are a leading light on the subject. This practice, unfortunately, is not uncommon and is termed coercive citation. It is poor referee practice, is unethical and is to be shunned. You may suggest that part of the paper has redundant data, or that it contains interesting but hidden data whose import the authors may not have realized. If you are unable to suggest some modifications or improvement, you are probably not assessing the papers adequately.
Will the readers be interested in the paper? Will the paper stimulate new thinking? Or change practice? Is the paper in the style required by the journal? Is the English scientific? Are the spellings, grammar and syntax correct?
You may wish to include personal comments to editor which you don't want the authors to see. These may advise the editor to seek another review, or obtain a statistical review or point out suspected research misconduct that may need investigation.
If you suspect plagiarism, inform the editor, but do specifically mention the plagiarised text and the source from which you suspect that it is copied from. Merely stating that there is plagiarism means that the editor has to search for the offending parts - a waste of time, since you are already aware of the details.
If the paper is good but is not relevant to the readership of the journal (eg. electron microscopic features of a disease, submitted to a general medical journal), you may suggest a few journals that the authors could target.
Specific questions to address for the various segments of an article are as follows -
Have the authors made their point clear? What is the aim of the research? Is the introduction too long? This is a common problem. Does it contain absolutely irrelevant material?
Materials and methods
Are the details sufficient? Are the dates correct? Are the definitions and criteria used scientific? Are the methods sufficient to enable another researcher to replicate the research? Are the hypothesis and the primary and secondary outcomes clear? Is there a protocol to compare the manuscript with, to ensure these have not been changed since the research was conducted? If the research is a clinical trial, was it registered? Does reporting adhere to the EQUATOR reporting recommendations for the particular study? Is the population studied appropriate to address the hypothesis? Or is there any selection bias? Are there any ethical issues? Has ethics committee consent been taken? Are the statistical methods appropriate? If you are not sure about this, suggest to the editor that the paper be seen by a statistician.
Do the numbers match? Do the percentages match the numbers? Do the dates (if stated) and data match? Have all relevant results been stated? Are the tables and figures appropriate and clearly labelled? Are there supplemental files that you need to review? Are there data in the results section, which are not referred to in the discussion? Are there lacunae in the data? If so, have these deficiencies been addressed in the discussion?
Is the discussion appropriate? Is it too long? Not uncommonly, the discussion is too brief. Are all issues mentioned in the discussion relevant to the data presented? Are there too many generalisations? Does the discussion flow well? Or do you think that transpositioning of some paragraphs would make the article read better? Are the arguments made in the discussion logical? Or have the authors read too much into their findings? Are there contradictions in the discussion? Has a major dissenting view on the subject not referred to so as to give a bias or slant to the paper? Do the results support the conclusions? Are the limitations discussed?
Is the paper well written? Is it a good scientific analysis? Is there a clear message? Do not blindly accept the argument or statement, “It is not uncommon to come across.....” or “First case of.....” or “has not been described before.....”. Check and crosscheck. The rarity of an entity may only represent a limited attempt on the part of the authors to identify previous cases.
What issues have not been referred to in the discussion. What are the lacunae? Is there any conflict of interest? In the current era of pharma-sponsored and pharma-supported articles as well as ghost-written articles, the referee (and editor) has to read between the lines of the article so as to ensure that no drug or technology is being pushed inordinately.
Once the main manuscript has been read and understood, it is time to review the rest of the paper.
Title, keywords and abstract
Is the title appropriate? Does it accurately represent the study? Can it be modified to make it more specific and to attract the readers' attention? The specificity may be scientific or geographic. Does it contain unfamiliar acronyms? Are the keywords, if asked for by the journal, appropriate? If not, suggest appropriate keywords. Does the abstract faithfully reflect the context, aim, methods, results and discussion? Are sample size and results quantified? Do the numbers match those in the body of the paper?
Tables and images
Are the tables really needed? If these merely duplicate what has been mentioned in the text – or vice versa – they need to be deleted. Sometimes, tables are easier to understand than text; in such cases, the text can be modified and only the tables can be retained. You, as referee, need to point this out. Are images needed to illustrate the point being made? Are these too many or too few? Are they in focus and is the exposure correct? Are higher magnifications or specific areas of the image needed? Do the legends match? Do the legends provide information that the reader can understand? Is there evidence of image manipulation?
Check the references. It is not uncommon to find that the authors have misinterpreted or misquoted a paper. If the central argument of the paper being reviewed hinges around this misunderstood point, the entire justification for the work may be flawed – and should lead to advice to reject the paper.
Are the references up to date? If there is no reference to the subject in the past few years, the topic is probably passé or that the author has not looked up the recent literature. Such a paper is likely to get rejected. Citations to pseudojournals or predatory journals are to be avoided.
Are there too many references? A letter or case report with 40 references would mean that the reference list is longer than the paper! In such cases, suggest that fewer references will be in order. Other papers may have too few references. Do make sure that all the relevant papers have been quoted. You may find that the authors have failed to cite previous data on the subject from the same – or a different - geographic area and with entirely different results. Citing mostly from one research group or one country – or ignoring a group – suggests a bias. If the authors quote their own previous paper, you must read the latter. If the new paper is more of the same or is an imalas publication (imalas – the reverse of salami in spelling and in meaning, wherein an author merely adds a few more cases to an older series and creates a new paper with an unchanged message), it does not deserve to be published.
Comments specific to Indian Journal of Cancer
Finally, this. As far as the Indian Journal of Cancer is concerned, while we are international – after all, science, scientists, their fora and disease recognise no boundaries, it is the Indian Journal of Cancer. Thus you have to think – would these data be of use to Indian physicians and researchers? The first Indian report of an entity which may have been documented abroad may be of interest. Data from India on treatment of a disease that is seen worldwide is of interest because of our different social structure and mores. In other words, act global but think local.
The checklist includes the most important points that the referee must address.
| » How to Write the Report|| |
Begin with a brief summary of the paper in a couple of sentences and then get down to the strengths and weaknesses of the paper. Your comments may be as brief as few sentences (if the paper is really bad and deserves outright rejection, or does not belong to the journal's field) or may be long.
Divide your comments into major and minor issues. The major ones are those that the authors must address. Minor ones refer to the occasional grammatical or spelling error. While it is the editor's decision to accept or reject the paper, you should offer your suggestion on what the editor should do, if that is not obvious in your comments.
A few points about the style of comments. Some referees send their comments only in lowercase letters, forgetting capitals. Others forget to use full-stops and commas or use atrocious grammar and spellings. The editor needs to correct these before forwarding the comments to the authors, leading to needless extra work. It is most jarring for an editor to read a comment that the “spellings and grammer (sic) needs (sic) to be improved”. It is not the referee's duty to correct every spelling or every grammatical error in the paper. You just need to point out that there are errors and mention a couple of examples.
Do not breathe a word about the paper to anyone – and remember not to use the data or hypothesis for your own research. Any violation of this is reviewer misconduct. All copies, print, as well as electronic, must be destroyed when the review is complete.
You will make occasional errors, letting a bad paper through, or suggest rejection of a good paper. The latter – rejecting a good paper – only means that the authors will publish it in another journal. But letting through a poorly researched paper has greater consequences as the data in such a published paper will be used by other investigators for their research and to guide patient care in future.
To end, if “reading,'riting and 'rithmatic” are important in school, reading, reviewing/refereeing are important for the researcher!
Acknowledgements and notes
The editor recused himself from the peer review process for this manuscript and a member of the editorial board handled the whole process offline.
The author is grateful to the two anonymous referees whose comments have vastly improved the quality of the manuscript.
| » References|| |
Stiller-Reeve M. How to write a thorough peer review. Career sheet. Nature 8 October 2018. doi: 10.1038/d41586-018-06991-0.
Annesley TM. Seven reasons not to be a peer reviewer – and why these reasons are wrong. Clin Chem 2012;58:677-9.
Paulus W. The referee who agrees to review and never responds again (NERO): A series of 37 cases of an emerging entity. Acta Neuropathologica 2018;35:647-8.
Pai SA. Medical journals – In the news and for the wrong reasons. Indian J Med Ethics 2014;11:7-9
Andrade C, Choudhury P. Do Indian researchers read Indian research? Indian J Psychiatry 1994;36:173-6.
] [Full text]