Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive

Abstract

Background: The evaluation of scholarly articles’ impact has been heavily based on the citation metrics despite the limitations of this approach. Therefore, the quest for meticulous and refined measures to evaluate publications’ impact is warranted. Semantic Scholar (SS) is an artificial intelligence-based database that allegedly identifies influential citations defined as “Highly Influential Citations” (HICs). Citations are considered highly influential according to SS when the cited publication has a significant impact on the citing publication (i.e., the citer uses or extends the cited work). Altmetrics are measures of online attention to research mined from activity in online tools and environments. Aims: The current study aimed to explore whether SS HICs provide an added value when it comes to measuring research impact compared to total citation counts and Altmetric Attention Score (AAS). Methods: Dimensions was used to generate the dataset for this study, which included COVID-19-related scholarly articles published by researchers affiliated to Jordanian institutions. Altmetric Explorer was selected as an altmetrics harvesting tool, while Semantic Scholar was used to extract details related to HICs. A total of 618 publications comprised the final dataset. Results: Only 4.57% (413/9029) of the total SS citations compiled in this study were classified as SS HICs. Based on SS categories of citations intent, 2626 were background citations (29.08%, providing historical context, justification of importance, and/or additional information related to the cited paper), 358 were result citations (3.97%, that extend on findings from research that was previously conducted), and 263 were method citations (2.91%, that use the previously established procedures or experiments to determine whether the results are consistent with findings in related studies). No correlation was found between HICs and AAS (r = 0.094). Manual inspection of the results revealed substantial contradictions, flaws, and inconsistencies in the SS HICs tool. Conclusions: The use of SS HICs in gauging research impact is significantly limited due to the enigmatic method of its calculation and total dependence on artificial intelligence. Along with the already documented drawbacks of total citation counts and AASs, continuous evaluation of the existing tools and the conception of novel approaches are highly recommended to improve the reliability of publication impact assessment.

The enhanced research impact of self‐archiving platforms: Evidence from bioRxiv

“…Altmetrics are metrics based on data generated by social networking platforms, aiming to provide insights into assessing the broader social impact of research beyond traditional citation‐based metrics (Bornmann, 2014). Previous studies have developed indicators based on usage data from social network platforms like Mendeley, Twitter, and others (Dardas et al, 2023; Glanzel & Gorraiz, 2015; Thelwall et al, 2023; Wang, Glänzel, & Chen, 2020; Zahedi et al, 2017). Altmetrics indicators are particularly suitable for this study as they quantify the extent to which readers take advantage of the free access provided by self‐archiving platforms.…”

“…Previous studies have developed indicators based on usage data from social network T A B L E 5 The effects of self-archiving preprint on citations in 0-3 years after publication. platforms like Mendeley, Twitter, and others (Dardas et al, 2023;Glanzel & Gorraiz, 2015;Thelwall et al, 2023;Zahedi et al, 2017). Altmetrics indicators are particularly suitable for this study as they quantify the extent to which readers take advantage of the free access provided by self-archiving platforms.…”

Mapping Two Decades of Research Productivity in the Middle Eastern and Arab Countries: A Comprehensive Bibliometric Analysis

“…Fostering research environments that promote interdisciplinary collaborations and encourage participation in global research networks, conferences, and publication outlets are also promising strategies to enhance the reach and impact of research outputs. It remains important to recognize that research impact is multifaceted and depends on various factors, including research quality, collaboration, interdisciplinary efforts, publication strategies, and engagement with the global research community [19]. The recommendations should be tailored to the specific context and goals of each region.…”

Erving Goffman: The Social Science Maverick. Assessing the Interdisciplinary Impact of the Most Cited American Sociologist

“…However, all competing search engines and citation analysis systems are hobbled by serious limitations (Aksnes, Langfeldt, and Wouters 2019; Bornmann and Daniel 2008; Dardas et al 2023; Yang and Meho 2007). Critics have argued, convincingly, that the time lag, negative evaluations, courtesy references, self-citations, and difficulty of assessing the reference context and standardizing units of analysis, compel the conclusion that “the citation count per se might not be a genuine and objective reflection of the publication’s scientific value” (Dardas et al 2023, 2). It is harder still to evaluate the publication’s impact across scholarly fields.…”

“…The SS index sorts out citations by relevance, distinguishing the “Highly Influential Citations” from the “Background” references and mentions marginal to the citer’s goals and research findings. However, the SS algorithm tracking the citation’s relevance has come under criticism in recent studies (Dardas et al 2023). A different classification is developed in this paper to assess Goffman’s impact.…”