信息资源管理学报 ›› 2021, Vol. 11 ›› Issue (6): 10-16,94.doi: 10.13365/j.jirm.2021.06.010

• 特稿 • 上一篇    下一篇

同行评议一致性研究

刘欣1 李江1 吴金闪2   

  1. 1. 南京大学信息管理学院,南京,210032; 
    2. 北京师范大学系统科学学院,北京,100000
  • 出版日期:2021-11-26 发布日期:2022-01-18
  • 作者简介:刘欣,硕士生,研究方向为信息计量学;李江(通讯作者),教授,研究方向为信息计量学,Email:lijiang@nju.edu.cn;吴金闪,教授,研究方向为科学学与系统科学。
  • 基金资助:
    本文系教育部科技发展资金项目“科技评价的国际比较研究”(2020ZL04)和国家自然科学基金面上项目“基于论文内容、学科概念地图和影响传递算法的论文评价方法研究”(72074030)的研究成果。

A Study of the Agreement of Peer Review

Liu Xin1 Li Jiang1 Wu Jinshan2   

  1. 1.School of Information Management, Nanjing University, Nanjing, 210032; 
    2. School of System Sciences, Beijing Normal University, Beijing, 100000
  • Online:2021-11-26 Published:2022-01-18

摘要: 评分一致性是衡量同行评议评价效果的重要指标之一。为了探究同行评议中评审人在研究质量评价上的一致性,本研究基于ICLR会议的公开同行评议数据,使用多位评审人对一项研究评分的方差衡量一致性,并采用单因素方差分析等方法,分析了对不同研究质量论文评审人评分一致性的差异。结果发现,同行评审对于质量较差论文的一致性高于质量较好的论文。基于这一证据,本研究提出以“择差”为目的的“底线评审”的构想,以完善当前以“择优”为宗旨的同行评议制度。

关键词: 同行评议, 共识程度, 一致性, 底线评审, 评优, 学术评价

Abstract: The agreement of reviewers’ ratings is one of the important aspects to measure the effectiveness of peer review evaluation. In order to explore the agreement of peer review in evaluating research quality, this study conducted an empirical analysis based on the open peer review data of the ICLR conference. We quantified the degree of the agreement as the variance of reviewer’s ratings on the evaluation of a single paper, and used one-way ANOVA to analyze the differences in the degree of consensus of reviewers on the papers of each score level. The results show that peer review has a higher degree of agreement on low-quality articles than high-quality articles. The conclusion of this study can be also extended to the review of research funding and provide theoretical basis for the evaluation strategy of the newly proposed “bottom-line review”.

Key words: Peer review, Degree of consensus, Consistency, Bottom-line review, Selection of the best, Academic evaluation

中图分类号: