Are methodological quality and completeness of reporting associated with citation-based measures of publication impact? A secondary analysis of a systematic review of dementia biomarker studies

Abstract

ObjectiveTo determine whether methodological and reporting quality are associated with surrogate measures of publication impact in the field of dementia biomarker studies.MethodsWe assessed dementia biomarker studies included in a previous systematic review in terms of methodological and reporting quality using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) and Standards for Reporting of Diagnostic Accuracy (STARD), respectively. We extracted additional study and journal-related data from each publication to account for factors shown to be associated with impact in previous research. We explored associations between potential determinants and measures of publication impact in univariable and stepwise multivariable linear regression analyses.Outcome measuresWe aimed to collect data on four measures of publication impact: two traditional measures—average number of citations per year and 5-year impact factor of the publishing journal and two alternative measures—the Altmetric Attention Score and counts of electronic downloads.ResultsThe systematic review included 142 studies. Due to limited data, Altmetric Attention Scores and electronic downloads were excluded from the analysis, leaving traditional metrics as the only analysed outcome measures. We found no relationship between QUADAS and traditional metrics. Citation rates were independently associated with 5-year journal impact factor (β=0.42; p<0.001), journal subject area (β=0.39; p<0.001), number of years since publication (β=-0.29; p<0.001) and STARD (β=0.13; p<0.05). Independent determinants of 5-year journal impact factor were citation rates (β=0.45; p<0.001), statement on conflict of interest (β=0.22; p<0.01) and baseline sample size (β=0.15; p<0.05).ConclusionsCitation rates and 5-year journal impact factor appear to measure different dimensions of impact. Citation rates were weakly associated with completeness of reporting, while neither traditional metric was related to methodological rigour. Our results suggest that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact.

Does quality of research in otolaryngology correlate with academic impact?

“…(Altmetrics is defined as the quantification of the digital footprint of a particular article that is not confined to the publishers and scientific community that is, social media, blogs and news outlets.) 10 This study is the first to examine the relationship between quality and impact in Otolaryngology. 12 Nieminen et al also did not find a significant relationship between the quality of psychiatric research among four journals and the total number of citations per article over 9 years.…”

“…Moreover, it should include the impact of the article on clinical practice 9 . However, as with “quality” there is no consensus definition of “impact.” 10 In fact, impact can also be thought of as a multifactorial concept that changes depending on the point of view. For example, on a larger scale impact can be directly related to the appropriate dissemination of research thereby effecting policy making.…”

“…(Altmetrics is defined as the quantification of the digital footprint of a particular article that is not confined to the publishers and scientific community that is, social media, blogs and news outlets. ) 10 …”

New quality and quantity indices in science (NewQIS): results of the first decade—project progress review

“…As already earlier critically discussed, they are not real measures for the quality of individual research (Pleger et al 2014). In this respect, a recent study addressed the question if methodological quality and completeness of reporting are associated with citation-based measures of publication impact (Mackinnon et al 2018). The authors performed a secondary analysis of a systematic review of dementia biomarker studies.…”

“…While citation rates were weakly associated with completeness of reporting, none of these metrics was related to methodological rigor. They suggested that high publication usage and journal outlet is not a guarantee of quality and readers should critically appraise all papers regardless of presumed impact (Mackinnon et al 2018). Therefore, qualitative aspects are better addressed by advanced meta-analysis approaches using i.e.…”