Peer assessment of CS doctoral programs shows strong correlation with faculty citations

Abstract

IMAGERY FROM SHUTTERSTOCK.COM counts of papers published in computer science journals and number of highly cited faculty). The final ranking is a weighted average of these measures. The scientometrics community criticized this approach because the choice of weights is not clearly justified. 4,6 The U.S. News ranking of doctoral programs in engineering b uses a weighted average of objective measures and subjective measures. As with the ARWU, justification for the ranking formula is lacking.Ranking of computer science doctoral programs published in 2010 by the U.S. National Research Council (NRC) 2 is notable for its effort to provide a justifiable ranking formula. The b https://www.usnews.com/education/bestgraduate-schools/articles/engineeringschools-methodology?int=9d0e08NRC collected objective measures and surveyed faculty to assess peer institutions on multiple measures of perceived quality. The NRC ranking group then built a regression model that predicts subjective measures based on the objective measures. The resulting regression model was used to provide ranking order. Unfortunately, the subjective and objective data collected during the NRC ranking project had questionable quality 3 and the resulting ranking did not find good reception in computer science community. c We find the NRC idea of calculating the ranking formula through regression modeling better justified than the alternatives. In this article, we address the data-quality issue that plagued the c http://www.chronicle.com/article/Too-Big-toFail/127212/ NRC ranking project by collecting unbiased objective data about programs in the form of faculty-citation indices and demonstrate that regression analysis is a viable approach for ranking computer science doctoral programs.We also obtain valuable insights into the relationship between peer assessments and objective measures. contributed articles ence departments. When people pages clearly separated such faculty from primary appointments, we included only the primary appointments in our list. When the people pages did not provide discriminable information about affiliations, we included all listed tenure-track faculty. The details of faculty selection for each university are in the "CS Department Data" file we maintain on our ranking webpage. f Overall, we collected the names of 4,728 tenure-track faculty members, including 1,114 assistant professors, 1,271 associate professors, and 2,343 full professors. Since assistant professors are typically only starting their academic careers and publication records, we treated them differently from associate and full professors, and for the rest of this article, we refer to associate and full professors as "senior faculty." Ranking DataThe distribution of program size is quite varied, with median faculty size of 22 positions, mode of 15, minimum of four, and maximum of 143 (CMU). The Pearson correlation between department size and USN CS score of the 119 programs ranked by U.S. News is 0.676, indicating larger departments are more likely to be higher ra...

Venue Analytics: A Simple Alternative to Citation-Based Metrics

“…Comparing citation counts with statistical significance requires an order-of-magnitude difference in the citation counts [30], which limits their utility in making fine-grained choices. Despite these quality issues, recent work [56] has demonstrated that citation-based metrics can be used to build a university ranking that correlates highly with peer assessment; we show that our method provides a similar quality of correlation.…”

“…We show that our venue scores correlate highly with other influence metrics, such as h-index [24], citations or highly-influential citations [54]. Additionally, we show that university rankings, derived from publication records correlate highly with both established rankings [17,37,42] and with recently published quantitative metrics [6,8,12,56]. arXiv:1904.12573v2 [cs.DL] 5 Jun 2019 2 RELATED WORK Quantitative measures of academic productivity tend to focus on methods derived from citation counts.…”

“…These include rankings curated by journalistic sources, such as the US News Rankings [37], the QS Rankings [42], Shanghai Ranking [43], Times Higher Education Rankings [17] and the National Research Council report [12]. In addition, we consider purely quantitative evaluation systems such as ScholarRank [56], CSRankings [6], CSMetrics [8], and Prestige Rankings [12]. We additionally include ScholarRank's t10sum across the matched faculty that our venue scores result uses.…”

“…We follow a recent paper [56], which demonstrated the efficacy of a citation-based metric in producing rankings with large correlation against US News rankings. We extend these experiments to include more baselines.…”

“…We extend these experiments to include more baselines. In contrast with [56], we use a rank correlation metric (namely Kendall's τ ), which naturally handles ordinal ranking systems. While ScholarRank [56] claimed a correlation of > 0.9 with US News, this was under Pearson's correlation coefficient, and the result under Kendall's τ is 0.768 in the published version and 0.757 using full precision ScholarRank.…”