How to write a review

Ulle Endriss — created: 16 November 2011 — last updated: 24 September 2017

On this page I make a number of suggestions for how to write a review for a paper submitted for publication to a journal or conference. Reviewing is an important part of every scientist's responsibilities and you'll most likely do a lot of it. (I have written hundreds of reviews, and I must have read thousands written by others, be it as the author of the paper under review, as a fellow reviewer, or as the editor commissioning the review.)

I should mention that my own background is in AI and I mostly review papers making a conceptual or mathematical contribution, and only very occasionally papers reporting on experimental work or describing systems. My views probably are somewhat influenced by this and certain conventions may well differ in other disciplines.

These notes have grown out of a set of instructions prepared for the students in my course on computational social choice in 2011.

Why?

Why do people write reviews? The first objective is to help select good and formally correct work for publication, so that others in the field can focus their attention on these (hopefully) high-quality papers. The second objective is to help the authors of the paper under review improve their work before it gets published.

Every now and then you will hear people say that reviewing is a thankless task and that there are no incentives for investing time and effort into writing high-quality reviews. This is a severe misunderstanding of reality. Of course, high-quality reviews benefit the research area you work in and thereby indirectly yourself, but the way you write your reviews also has a much more immediate impact on your career. While your identity will not be known to the authors, it usually is known to several (typically senior) colleagues: editors, PC chairs, senior PC members, and fellow reviewers. The views of all of these people regarding your professional qualities as a scientist will be influenced by your review.

The Opening Paragraph

It is usually a good idea to start your review with a paragraph summarising the most significant contributions of the paper under review. The main purpose of this is to show to the authors (as well as to the person who has commissioned your review and possibly your fellow reviewers) that you have actually read the paper and made a sincere attempt at understanding it.

You may not always agree with the authors about what the main contributions actually are and how they should be interpreted. Making this point clear is a second important function performed by your introductory paragraph.

It is important to use your own words when writing this summary (otherwise the summary cannot fulfil either one of the above two functions). You should definitely not just copy a few key sentences from the introduction written by the authors (that's just a waste of virtual paper).

Structuring the Body of the Review

One approach is to structure the body of your review in terms of the review criteria below.

A second, in my view often preferable, approach is to organise the comments you wish to make in terms of their importance. Begin with criticism aimed at the overall approach taken, then discuss specific technical points, and conclude with remarks on presentation quality and (possibly) a (short) list of typos and similar minor issues.

Review Criteria

It is common practice to assess a paper under review according to the criteria listed below. They are described in more detail in my notes on how to write a paper.

Professional Conduct