How to write a review

Ulle Endriss — created: 16 November 2011 — last updated: 30 September 2023

On this page I make a number of suggestions for how to write a review for a paper submitted for publication to a journal or conference. Reviewing is an important part of every scientist's responsibilities and you'll most likely do a lot of it. (I have written hundreds of reviews, and I must have read thousands written by others, be it as the author of the paper under review, as a fellow reviewer, or as the editor commissioning the review.)

I should mention that my own background is in AI and I mostly review papers making a conceptual or mathematical contribution, and only very occasionally papers reporting on experimental work or describing systems. My views probably are somewhat influenced by this and certain conventions may well differ in other disciplines.

These notes have grown out of a set of instructions prepared for the students in my course on computational social choice in 2011. In September 2023 I gave a talk on the topic at the Doctoral Consortium of the 26th European Conference on Artificial Intelligence (ECAI). The slide deck includes additional points not covered here.

Why?

Why do people write reviews? The first objective is to help select good and formally correct work for publication, so that others in the field can focus their attention on these (hopefully) high-quality papers. The second objective is to help the authors of the paper under review improve their work before it gets published.

Every now and then you will hear people say that reviewing is a thankless task and that there are no incentives for investing time and effort into writing high-quality reviews. This is a severe misunderstanding of reality. Of course, high-quality reviews benefit the research area you work in and thereby indirectly yourself, but the way you write your reviews also has a much more immediate impact on your career. While your identity will not be known to the authors, it usually is known to several (typically senior) colleagues: editors, PC chairs, senior PC members, and fellow reviewers. The views of all of these people regarding your professional qualities as a scientist will be influenced by your review.

By emphasising this fact I do not mean to say that you should necessarily accept every single review request. Just do your fair share as a member of the research community you belong to. Most importantly, once you have accepted to review a given paper, do a professional job of it. When deciding which requests to accept, you may want to take into account criteria such as these: Is this a conference/journal you know to be serious? Do you (try to or hope to) publish there every now and then yourself? Is it a socially responsible enterprise (say, a community-driven open-access journal or a conference with reasonable registration fees)?

The Opening Paragraph

It is usually a good idea to start your review with a paragraph summarising the most significant contributions of the paper under review. The main purpose of this is to show to the authors (as well as to the person who has commissioned your review and possibly your fellow reviewers) that you have actually read the paper and made a sincere attempt at understanding it.

You may not always agree with the authors about what the main contributions actually are and how they should be interpreted. Making this point clear is a second important function performed by your introductory paragraph.

It is important to use your own words when writing this summary (otherwise the summary cannot fulfil either one of the above two functions). You should definitely not just copy a few key sentences from the introduction written by the authors (that's just a waste of virtual paper).

Structuring the Body of the Review

One approach is to structure the body of your review in terms of the review criteria below.

A second, in my view often preferable, approach is to organise the comments you wish to make in terms of their importance. Begin with criticism aimed at the overall approach taken, then discuss specific technical points, and conclude with remarks on presentation quality and (possibly) a (short) list of typos and similar minor issues.

Review Criteria

It is common practice to assess a paper under review according to the criteria listed below (they are also discussed in my notes on how to write a paper):

• Relevance. Is the topic of the paper relevant to the conference or journal it has been submitted to? In most cases, at least if the paper is otherwise reasonable, the answer will be yes. I recommend to be lenient on this criterion and, if necessary, to give the authors the benefit of the doubt. You don't want to accidentally cut off a new and possibly exciting direction of research, just because you yourself are not quite ready for it yet.
• Significance. Are the results reported important? Will others be interested? Will their own work be influenced by this paper? If the authors do not make it clear why their results should be considered significant, then you should feel free to evaluate the paper as scoring low with respect to significance. Alternatively, if you happen to understand why the results are significant, but the authors were not able to communicate this well in their paper, then you should help them and suggest appropriate revisions.
• Originality. Does the paper include genuinely new ideas? This could be a novel type of question to ask or a novel approach for obtaining results in an existing subfield. Don't expect every single paper to bring about a scientific revolution. Most papers will be "incremental" (a word that too often is used in a derogatory manner) and present some modest progress along a well established path of investigation. This is exactly what we should hope for! (A research area that's always jumping from one revolutionary idea to the next is not a serious one.) When you review a paper and you believe that a result claimed to be new by the authors in fact is not new, then it is crucial that you provide full bibliographic references to that result in your review. Simply stating "this is a known result" is a clear sign for an unprofessional review.
• Technical soundness. This is not just about the formal correctness of theorems and their proofs, but also, for instance, about the mathematical precision of definitions. When assessing the merits of a paper, you are not expected to do the work of the authors for them. For example, if a proof appears unnecessarily difficult, then you should say so and clearly state that you have not been able to verify the result. More generally, if a particular result is claimed, but insufficient details are given for a reviewer to verify the claims, then this reflects poorly on the technical soundness of the paper. Of course, a paper with serious technical flaws should not be accepted. Having said this, not every minor inaccuracy immediately has to lead to rejection. Indeed, most journals allow for the possibility of accepting a paper subject to the authors making certain revisions of their work. But even for conferences (which usually allow for only a single round of reviewing), if you can suggest a clear-cut fix to a problem and the rest of the paper shows that the authors are competent, then you can probably trust them with resolving the issue when they prepare the camera-ready version of their paper.
• Scholarship. Do the authors give due credit to related work? Avoid suggesting that your own work should have been cited (it usually is difficult to be really objective about such matters). Trying to up your own citation count by pushing authors you review to cite you is clearly unethical. If a required reference to your own work is missing, probably one of the other reviewers will catch it, so you don't need to worry about this.
• Clarity. Is the material described easy to follow and well structured? I find this an extremely important criterion and wish that people would pay a little more attention to it. If a paper is otherwise strong, a poor score on clarity is probably not enough to get a paper rejected, but is is a real shame, not least for the authors themselves. So try to give them some hints for how they can do better in getting their message across. If the authors did a great job, worked hard on coming up with a readable notation and a consistent terminology, simplified all their proofs as much as possible, and were honest about all the debts owed to prior work by others, you should reward such exemplary behaviour rather than punish it. That is, don't confuse "easy to understand" with "trivial".
• Presentation. This covers the very basics of academic writing, such as orthography, grammar, formatting issues, and the usefulness of figures. Ideally, a paper should have no typos. If you still find a handful, as you probably will, it is nice if you list them at the end of your review. If the authors obviously didn't bother running a spell-checker before submission, you can just say so and ask them to change their evil ways. For grammatical errors, matters are a bit more complex. You cannot expect a nonnative speaker of English to deliver a perfectly formulated manuscript in English. What matters here is whether grammatical issues truly hinder readability and understanding. Try to help the authors to eliminate the most glaring mistakes. Of course, if things are so bad that you cannot understand what the authors are trying to say, then you will not be in a position to certify that the paper is significant, original, or technically sound.

Professional Conduct

• Politeness. While you should be critical, the tone of your review must always remain polite and professional. Your goal should be to help the authors to improve their paper as much as possible, independently of whether it will get accepted or rejected on this particular occasion.
• Collegiality. In writing your review, you are helping the editor of the journal or the PC chair of the conference to select the best papers and/or to ensure that only papers that meet the standards of this particular publication venue are accepted. But keep in mind that you are not making these decisions yourself: for any serious publication venue, you will be one of several reviewers and the final decision will be taken by yet another person. Therefore, as a reviewer, in your comments for the authors, you should never include an explicit remark saying "this paper should be accepted" or "this paper should be rejected" (of course, you can—and usually are asked to—make this kind of statement confidentially).
• Confidentiality. Reviewing for conferences and journals is usually blind (and often double-blind). This means an author does not know the identity of the reviewer (and for double-blind reviewing the reviewer does not know the identity of the author either). You should respect these conventions and not attempt to find out about the identities of the authors for a paper submitted for double-blind review or reveal your own identity to an author whose paper you have reviewed in the past. The former convention is intended to avoid well-established authors getting treated more favourably than others. The purpose of the latter is to ensure that reviewers are free from social pressures when giving their assessment, which ultimately serves the integrity of the field. Both conventions are vital for scientific progress.
• Timeliness. Respond promptly to review requests and invitations to join programme committees and the like, particularly if you are going to decline. Return your review by the deadline you agreed to when you signed up for the job. For conferences, in particular, this is an absolute must: if you are late, most likely someone else will have to do your work for you, at the very last minute. For a journal review it is usually not a major problem to ask for an extension of, say, a week—provided you do so a few days before the deadline agreed upon originally.