Feasibility of common bibliometrics in evaluating translational science

Abstract

IntroductionA pilot study by 6 Clinical and Translational Science Awards (CTSAs) explored how bibliometrics can be used to assess research influence.MethodsEvaluators from 6 institutions shared data on publications (4202 total) they supported, and conducted a combined analysis with state-of-the-art tools. This paper presents selected results based on the tools from 2 widely used vendors for bibliometrics: Thomson Reuters and Elsevier.ResultsBoth vendors located a high percentage of publications within their proprietary databases (>90%) and provided similar but not equivalent bibliometrics for estimating productivity (number of publications) and influence (citation rates, percentage of papers in the top 10% of citations, observed citations relative to expected citations). A recently available bibliometric from the National Institutes of Health Office of Portfolio Analysis, examined after the initial analysis, showed tremendous potential for use in the CTSA context.ConclusionDespite challenges in making cross-CTSA comparisons, bibliometrics can enhance our understanding of the value of CTSA-supported clinical and translational research.

Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study

“…Although citation tracking and analysis provided as part of the services of Scopus, WoS, and other citation databases can directly answer questions about publication output and suggest latent impact, several complications emerge when attempting to put the citation metrics into context. First, without a meaningful comparison, there is no basis for assessing this metric as representing relative success/failure in the generation of scientific output [4]. Second, publication counts offer a very limited reflection of scientific output.…”

“…Evaluation in this context relies on a diverse set of process and outcome metrics meant to capture a broad range of activities and dimensions of successful translational science [4]. These activities may include effective training of translational scientists; development of tools and methods that open new fields of inquiry; clinical trials support services to increase efficiency and start-ups; research support of collaboration; and the impact and influence of supported researchers [3].…”

“…The recent advancements of bibliometrics resources and information technology allow us to shift from basic bibliometrics (process-level measures) such as publications or citation counts to field-and time-normalized "comparative" approaches (impact-level measures) such as snowball metrics, benchmarking, and outputs in top percentiles [14,15]. Research-supporting institutions are now able to gain a more nuanced picture of the citation impact of research than can be captured in raw citation counts alone [4,16]. One of CTSA consortium-led evaluation workgroups has explored the feasibility and utility of both basic and improved bibliometric approaches.…”

“…One of CTSA consortium-led evaluation workgroups has explored the feasibility and utility of both basic and improved bibliometric approaches. Workgroup members concluded that bibliometric analysis contributes to the evaluation of CTSA-supported clinical and translational research as an important complementary method [4,17].…”

Applying Data Analytics and Visualization to Assessing the Research Impact of the Cancer Cell Biology (CCB) Program at the University of North Carolina at Chapel Hill

“…In addition, researchers generally agree that bibliometric analysis generates objective, sophisticated, quantitative measures of research impact for scientific outputs (Agarwal et al 2016;Rosas, Kagan and Schouten 2011). Even federal funding agencies such as the National Institutes of Health (NIH) have mandated publication tracking and reporting as a component for grants renewal (Schneider et al 2017). Overall, bibliometric analysis helps both researchers and organizations demonstrate their research capacity, capabilities, and competency.…”