Abstract: The agreement of reviewers’ ratings is one of the important aspects to measure the effectiveness of peer review evaluation. In order to explore the agreement of peer review in evaluating research quality, this study conducted an empirical analysis based on the open peer review data of the ICLR conference. We quantified the degree of the agreement as the variance of reviewer’s ratings on the evaluation of a single paper, and used one-way ANOVA to analyze the differences in the degree of consensus of reviewers on the papers of each score level. The results show that peer review has a higher degree of agreement on low-quality articles than high-quality articles. The conclusion of this study can be also extended to the review of research funding and provide theoretical basis for the evaluation strategy of the newly proposed “bottom-line review”.

Key words: Peer review, Degree of consensus, Consistency, Bottom-line review, Selection of the best, Academic evaluation