Abstract: To investigate the effectiveness of Altmetrics in measuring societal impact, this paper carried out some quantitative analysis and verification, with 519893 papers from the international popular citation databases as samples. Based on Web of Science, our article visualized the research papers on Altmetrics from 2010 to 2020 with the help of VOSviewer. The visualization result has demonstrated the intersection between Altmetrics and societal impact. In parallel, correlation analysis，partial correlation analysis and principal component analysis were undertaken by Stata16 among RCR, CNCI, Dimensions, Mendeley, Twitter and times cited to interpret the implications of Altmetrics indicators, and then Altmetrics was divided into citation-like indicators(i.e.,Mendeley) and attention indicators(i.e., Twitter). The results show that Altmetrics deviates from the goal of societal impact. Mendeley is included in the citation-like indicators with a principal component load of 0.76, indicating that it can be used as an early substitute indicator for papers’ scientific impact. The lack of construct validity in the current studies indicates the absence of theoretical basis in terms of the application of Altmetrics into societal impact measuring, and it also means that the case study method remains the favored one.

Key words: Altmetrics, Societal impact, Research evaluation, Bibliometrics, Case study