Peer review is at the core of what defines scholarship and thus academia itself. Yet for the most part, no one ever teaches scholars how to write a helpful peer review report for journals, publishers, or conference programming committees.
Often our only insight into what a review should be are the reviews we ourselves get. And if we never get positive or helpful reviews, we tend to think of the review process as combative, idiosyncratic, and a practice in gatekeeping.
But not all critical reviews are the product of the nefarious Reviewer 2, out to sabotage our careers and make us feel bad.
We can use reviewing to push our anonymous colleagues to present the best versions of their work. Doing so requires we approach reviewing with respect and care, recognizing there is a human behind the work.
So how can you produce critical but non-jerky reviews? Read on.
Check the existing review criteria
If the journal, publisher, or conference committee didn’t send any criteria, ask if they have a standard list of what they want you to focus on. If they don’t, make a list of things you will assess.
Like a grading rubric, this criteria will keep you focused, make your feedback easier to comprehend, and allow you to go back at the end and honestly weigh the merits and weaknesses of the submission.
Get a sense of the paper on its own terms
Next, skim the paper to get a general sense of what it is about. Read the abstract and review the section subheads. Get a sense of the shape of the paper.
Then read through and take notes under each of the review criteria on your list.
“Academic writing feels fun again” — IoF author Heath Fogg Davis
Take the stress out of publishing and see why so many interdisciplinary scholars trust Ideas on Fire to edit their books and journal articles.
What to evaluate in your peer review report
Below is my go-to criteria for conference papers and journal articles, but the specific categories you use may differ based on your field and the specific manuscript.
Introduction (topic/significance)
What is this project about? Is it an original contribution to the field? A new take on a problem? What will be the big takeaway from this paper? All of these are questions the author should be answering clearly in the introduction. The introduction should make the case that this text offers something new, and you are judging how well the author supports that claim in the manuscript.
If you know based on other reading you’ve done that the text you’re reviewing is not very original or significant, then you should say that as it is something the author should look into. If you are not aware of existing work that has dealt with the text’s problem in this specific way though, don’t assume that research exists somewhere and the author missed it. That is a jerky Reviewer 2 move and best avoided.
Thesis/argument
What big, overarching point is this text trying to make? Both the abstract and the introduction should state this clearly. If you can’t discern an argument from either, that’s a bad sign. If after reading the whole manuscript you still don’t know their argument, that’s an even worse sign. Texts without arguments should not be published—they’re not ready yet. If you can identify the argument, does the evidence provided support it? Depending on the paper’s field(s), such evidence might be provided via data, analysis, or theory.
Each section of the paper should have internally supported claims: Does the literature review support the research questions and hypotheses? Does the discussion of methods support the choices made, the archive/data source selected, and the approach to analysis? Does the analysis seem logical and supported with data, and do the conclusions follow from that analysis?
Literature review and theoretical soundness
Depending on the manuscript’s field(s), theory and the literature review may play different roles and appear in different sections. Does the literature cited support the author’s topic, argument, stakes, methodology, analysis, and conclusions? Has the author done their due diligence in citing and engaging relevant literature for the topic? Is there work they should have cited or that you think would benefit their analysis? Does the lit review situate the paper in a specific field or subfield, or make interdisciplinary connections? Does the author adequately explain the literature cited? Do you think they misinterpreted anything from the work they are citing? (If they cited you, did they misinterpret you? It’s more common than you might think.)
Your level of knowledge of a topic of a manuscript assigned to you will vary, so focus first on what the author thinks is significant and if that seems convincing to you. Then, generously, suggest other texts and fields they might bring into conversation with their project. It is not a failed project if they have not read everything you have, but it might need major revision if they are not engaging with specific scholarly conversations that you know are relevant. And sometimes, yes, they should have cited you. But if you don’t want to be “that reviewer,” also suggest other people who do work like yours if they failed to cite them as well.
Methods and methodological soundness
Did what the author do seem like a good way to address their topic or research questions? Note that this question does not mean Is this the best or only way to study that topic? Nor does it mean How would you—the reviewer—have done the study differently? Instead, consider if the author proceeded with their project in a way that could actually answer what they were trying to answer. Each method can only answer certain questions. Are their methods and research questions aligned?
Also, does the author explain and support their chosen method—within the norm of their discipline(s) or interdisciplines?
There is little an author can do if their project design is fundamentally flawed, except go do a different project. If an author’s research is not sound or is unethical, it is important that you point that out to them and the journal editors, publishers, or conference program committee.
Findings and analysis
Does the analysis follow from the data collected? Are the author’s conclusions supported by that data and analysis? Does the author make a case for what their findings contribute to the bodies of literature they used to frame their manuscript? These are all things their text should do, so take care to include this evaluation in your review.
Conclusion
Does the author’s conclusion follow from their findings and analysis? Do they connect their argument back to the broader theory/literature? Do they acknowledge any limitations or do they overstate their findings? Do they make a convincing claim for the significance of those findings in terms of contributing to what is known about their topic?
Structure and writing
The final thing you’ll evaluate the text on is writing. It is important that you do not copyedit the text—it’s not your job, it takes way longer than you should be spending reviewing an article, and the manuscript will get a professional copyedit anyway later if it gets published. However, if there are common or glaring errors, indicate that and cite a few examples so they can be fixed later.
Also, don’t worry about the citation style (that’s the job of the publisher, journal editor, or conference program committee).
You do want to focus on the overarching clarity and structure of the manuscript (a very light version of what professional editors call developmental editing). Does it build logically? Does it have proper sign posting for how each section relates to the whole? Can you understand what the author is trying to say in every paragraph?
And finally, never assume the language the paper is written in is not the authors’ own (another jerky Reviewer 2 move).
Concluding thoughts
When conducting a peer review report, whether on a book, article, or conference paper, always remember that you are assessing the manuscript on the grounds of what the author was trying to do. Avoid judging out of hand the soundness of the topic, methods, and theory they chose. If you think their topic is not worthy of study, you should decline to do the review. If you would have studied it a different way, then go study it that way. You instead should focus on how well the author supports their argument, methods, analysis, and findings. That’s the basis of a great peer review report.
Share this:
